Long-Context Fine-Tuning with Limited VRAM
Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive. We combine Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage. Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, and HGA loads a bounded set of exact historical tokens for each query block. On Qwen3-8B with 4-bit QLoRA and PG19, dense training on a 16 GB Quadro RTX 5000 fits 2,048 tokens but fails at 4,096, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM. Under evalu
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 64%Breakthrough in long-context efficiency announced →
- PossiblePossibly related (embedding) · 55%Biggest, baddest model to fill 144GB VRAM + 120GB RAM to the brim, regardless of speed →
- PossiblePossibly related (embedding) · 54%I mapped which local LLMs actually fit each RAM tier, 8 to 128GB (open dataset) →
- PossiblePossibly related (embedding) · 52%Tesla V100 16GB local LLMs, single and dual NVLink benchmarks →
- PossiblePossibly related (embedding) · 51%Devs - you have 64gb of VRAM - which model do you use for coding? →
- LinkedLinked via arxiv author · 85%Vladimir Fedosov →
“Long-Context Fine-Tuning with Limited VRAM”
- LinkedLinked via arxiv author · 85%Aleksandr Sazhin →
“Long-Context Fine-Tuning with Limited VRAM”
- LinkedLinked via arxiv author · 85%Artemiy Grinenko →
“Long-Context Fine-Tuning with Limited VRAM”