Loop the Loopies!
We present Loopie, the most powerful looped Transformer to date. The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6Bparameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 52%Build your first transformer from scratch →
- PossiblePossibly related (embedding) · 46%H64LM: A 249M-parameter Mixture-of-Experts Transformer built from scratch in PyTorch [P] →
- FuzzyOverlapping authors or contributors · 62%TauricResearch/TradingAgents →
“Shared author/contributor keys: xiao”
- FuzzyOverlapping authors or contributors · 62%sgl-project/sglang →
“Shared author/contributor keys: zhou”
- LinkedLinked via arxiv author · 85%Zitian Gao →
“Loop the Loopies!”
- LinkedLinked via arxiv author · 85%Yilong Chen →
“Loop the Loopies!”
- LinkedLinked via arxiv author · 85%Yihao Xiao →
“Loop the Loopies!”
- LinkedLinked via arxiv author · 85%Xinyu Yang →
“Loop the Loopies!”