ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce this bottleneck by either imposing sparsity, so that each query attends to only a small subset of keys, or by using low-rank/kernel sketches, so that global interactions are compressed into a lower-dimensional representation. We propose \emph{ELSAA}, an efficient low-rank and sparse approximation of attention. Importantly, ELSAA does \emph{not} decompose the learned projection or output matrices of the Transformer int
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 55%H64LM: A 249M-parameter Mixture-of-Experts Transformer built from scratch in PyTorch [P] →
- PossiblePossibly related (embedding) · 52%Transformer →
- PossiblePossibly related (embedding) · 51%Training transformers where every layer W = V·Uᵀ from initialization reveals a corpus-determined optimal rank - looking for arXiv endorser (cs.LG) [D] →
- PossiblePossibly related (embedding) · 50%Transformers in Deep Learning: How Self-Attention Changed Modern AI - Snowflake →
- FuzzySimilar title/name (fuzzy) · 87%lucidrains/x-transformers →
“Fuzzy title match (0.94): “ELSAA: Efficient Low-Rank and Sparse Attention Approximation” ≈ “lucidrains/x-transformers””
- FuzzySimilar title/name (fuzzy) · 84%huggingface/transformers →
“Fuzzy title match (0.92): “ELSAA: Efficient Low-Rank and Sparse Attention Approximation” ≈ “huggingface/transformers””
- LinkedLinked via arxiv author · 85%Mahdi Heidari →
“ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers”
- LinkedLinked via arxiv author · 85%Mohammad Mahdi Rahimi →
“ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers”