Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability
Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is underexploited by the conventional matrix-centric view. Tensor decompositions and tensor networks provide a principled algebraic language for this structure, yet the literature often treats them as isolated compression mechanisms. This survey organizes tensor methods for LLMs through two complementary views: a seven-stage lifecycle taxonomy covering tokenization, embeddings, pre-training, adaptation,
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 60%Transformer →
- PossiblePossibly related (embedding) · 54%Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers →
- FuzzySimilar title/name (fuzzy) · 84%xorbitsai/inference →
“Fuzzy title match (0.92): “Tensor Methods for Language Models: From Token Representatio” ≈ “xorbitsai/inference””
- LinkedLinked via arxiv author · 85%Matvei Tarasov →
“Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Inter”
- LinkedLinked via arxiv author · 85%Salman Ahmadi-Asl →
“Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Inter”
- LinkedLinked via arxiv author · 85%Andre L. F. de Almeida →
“Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Inter”
- LinkedLinked via arxiv author · 85%Andrzej Cichocki →
“Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Inter”