Approximate Muon with low-rank adapters
The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses ma
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- LinkedLinked via arxiv author · 85%Ben Anson →
“Approximate Muon with low-rank adapters”
- LinkedLinked via arxiv author · 85%Conor Houghton →
“Approximate Muon with low-rank adapters”
- LinkedLinked via arxiv author · 85%Edward Milsom →
“Approximate Muon with low-rank adapters”