Apparently you can skip entire transformer blocks at load time with minimal performance impact
Following recent (very cool) papers, I implemented this as a --skip-layers flag to a llama.cpp fork, so it just never instantiates the blocks you tell it to skip. Bake-time pruning already exists (--prune-layers, mergekit passthrough etc.); this is just the runtime version of the
Why it matters
This story from Reddit r/LocalLLaMA is relevant to the Open Source branch of the AI ecosystem and may affect models, products, or research direction.
Technical breakdown
Following recent (very cool) papers, I implemented this as a --skip-layers flag to a llama.cpp fork, so it just never instantiates the blocks you tell it to skip. Bake-time pruning already exists (--prune-layers, mergekit passthrough etc.); this is just the runtime version of the same idea. One important note here is that which blocks you skip matters by orders of magnitude, so i had to ship it wi
Business impact
Watch for product launches, funding moves, or policy shifts tied to this headline.