AVQ-Attention: Adaptive Vector-Quantized Attention
The $\mathcal{O}(N^2)$ complexity of attention over $N$ tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to $\mathcal{O}(MN)$ by representing keys with $M$ codewords, but applies uniform codebook capacity regardless of where attention mass concentrates: high-attention regions of key space may be coarsely approximated while low-attention regions waste representational capacity. We propose Adaptive Vector-Quantized (AVQ) Attention, which adaptively allocates codebook capacity based on attention importance. Starting from a small set of
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- LinkedLinked via arxiv author · 85%Patrick Forré →
“AVQ-Attention: Adaptive Vector-Quantized Attention”
- LinkedLinked via arxiv author · 85%Amir Habibian →
“AVQ-Attention: Adaptive Vector-Quantized Attention”
- LinkedLinked via arxiv author · 85%Yuki M. Asano →
“AVQ-Attention: Adaptive Vector-Quantized Attention”
- LinkedLinked via arxiv author · 85%Max Welling →
“AVQ-Attention: Adaptive Vector-Quantized Attention”
- LinkedLinked via arxiv author · 85%Winfried van den dool →
“AVQ-Attention: Adaptive Vector-Quantized Attention”