Rethinking Routing Exploration in Modern Mixture-of-Experts
Modern mixture-of-experts models combine finer expert segmentation with experts shared across all tokens. We show that these choices create an exploration dilemma: segmentation expands the routing space, while shared experts reduce the variation needed to explore it effectively. MirrorMoE addresses this mismatch by adaptively suppressing unnecessary experts for each token. Experiments across model scales show improved pretraining and downstream performance, allowing shared-expert models to better exploit their available capacity.