Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additio
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 55%Adaptive Mixture of Experts Gate (AMG) [R] →
- FuzzyOverlapping authors or contributors · 62%google-research/google-research →
“Shared author/contributor keys: sun”
- LinkedLinked via arxiv author · 85%Simeng Sun →
“Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration”
- LinkedLinked via arxiv author · 85%Roger Waleffe →
“Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration”