Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper investigates Manifold-Constrained Hyper-Connections (mHC), a generalisation of residual connections, as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules. We find that mHC can finetune frozen Transformers, but that its role differs fundamentally from the original pre-training setting: in finetuning, fixing the residual mixing matrix to identity often improves performance. As a
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 50%H64LM: A 249M-parameter Mixture-of-Experts Transformer built from scratch in PyTorch [P] →
- LinkedLinked via arxiv author · 85%Valentijn Oldenburg →
“Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning”
- LinkedLinked via arxiv author · 85%Floris de Kam →
“Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning”
- LinkedLinked via arxiv author · 85%Bente Zuijdam →
“Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning”
- LinkedLinked via arxiv author · 85%Lieve Eberson →
“Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning”
- LinkedLinked via arxiv author · 85%Nicky van Zutphen →
“Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning”
- LinkedLinked via arxiv author · 85%Stef de Wildt →
“Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning”
- LinkedLinked via arxiv author · 85%Ivo Verhoeven →
“Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning”