Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass. On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slice
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 54%Transformer →
- PossiblePossibly related (embedding) · 52%Fine-tuning →
- LinkedLinked via arxiv author · 85%Mohammed Sabry →
“Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model”
- LinkedLinked via arxiv author · 85%Sean Augenstein →
“Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model”
- LinkedLinked via arxiv author · 85%Keith Rush →
“Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model”
- LinkedLinked via arxiv author · 85%Lucio Dery →
“Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model”