Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are c
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 56%Transformer →
- PossiblePossibly related (embedding) · 56%engineering87/llm-atlas →
- PossiblePossibly related (embedding) · 52%Transformers in Deep Learning: How Self-Attention Changed Modern AI - Snowflake →
- PossiblePossibly related (embedding) · 50%amitshekhariitbhu/llm-internals →
- LinkedLinked via arxiv author · 85%Tiberiu Musat →
“Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks”
- LinkedLinked via arxiv author · 85%Tiago Pimentel →
“Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks”
- LinkedLinked via arxiv author · 85%Nicholas Zucchet →
“Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks”
- LinkedLinked via arxiv author · 85%Thomas Hofmann →
“Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks”