Sliding-window beats linear attention
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable. Several alternatives have been proposed to fix the quadratic scaling problem, one of which is retrofitting LLMs to use Linear Attention. This idea has attracted a lot of attention, given its promise to solve the quadratic scaling problem with state-of-the-art performance at low cost. However, this line of research has not been pr
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 51%Breakthrough in long-context efficiency announced →
- PossiblePossibly related (embedding) · 50%New Server Hopes to Break Through AI’s “Memory Wall” →
- PossiblePossibly related (embedding) · 49%Looking for feedback on a small test SLM I built completely from scratch [P] →
- PossiblePossibly related (embedding) · 49%How can we solve long-range recall in linear attention? [D] →
- PossiblePossibly related (embedding) · 48%Matrix Orthogonalization Improves Memory in Recurrent Models →
- LinkedLinked via arxiv author · 85%Alexia Jolicoeur-Martineau →
“Sliding-window beats linear attention”
- LinkedLinked via arxiv author · 85%Rhea Sanjay Sukthanker →
“Sliding-window beats linear attention”
- LinkedLinked via arxiv author · 85%Pashmina Cameron →
“Sliding-window beats linear attention”