The same residual-codebook idea from the Phase 1 embedding compression, applied to a completely different, much larger part of the model: all 78 MLP weight matrices (gate/up/down projections across all 26 layers, ~1.66B params — 63.6% of the whole model), each compressed independently.
A feasibility check on this project's own similarity-analysis data showed literal cross-layer codebook sharing doesn't survive at any reasonable quality bar for these weights — so this is independent compression instead: every one of the 78 matrices gets its own codebook, the same core mechanism as the embedding, just repeated 78 times rather than shared once. Attention (q/k/v/o) and the embedding are left untouched at bf16 — only the MLP.
The very first attempt used a fixed K=1024 codebook size for every matrix and only varied the number of subspaces — but the codebook-overhead math meant down_proj (4x larger than gate/up_proj) paid 4x the fixed codebook cost at the same K, landing at 16.72 bits/param — worse than doing nothing (bf16 is 16 bpp). This page serves the fixed version: K is chosen per-matrix to target 10 bits/param, achieving a real 9.44 bpp average instead.
Every variant answered the identical 200 held-out questions, judged the same blinded way. Storage/memory/speed were measured in one consistent benchmark (same GPU, same prompts, same warmup/sync discipline) so these numbers are directly comparable to each other, not just to baseline.
| variant | grounded acc. | open acc. | grounding rate |
|---|---|---|---|
| Uncompressed (baseline) | 92.9% | 80.0% | 93.5% |
| RQ (embedding) | 92.9% | 86.7% | 93.5% |
| NF4 (weights) | 91.6% | 86.7% | 94.2% |
| LUT (serving strategy) | 92.3% | 84.4% | 93.5% |
| Combined RQ+NF4 | 91.0% | 80.0% | 94.2% |
| MLP RQ v2 (this page) | 92.3% | 84.4% | 93.5% |
| variant | storage | peak GPU memory (generation) | speed |
|---|---|---|---|
| Uncompressed | 5.28 GB | 5,278.8 MB (1.0x) | 46.65 ms/tok (1.0x) |
| RQ | 412 MB embedding (vs. 1,180 MB bf16) | 5,278.8 MB (1.0x — no runtime savings) | 46.57 ms/tok (1.0x) |
| NF4 | ~2.2 GB whole model | 3,533.2 MB (0.67x) | 80.32 ms/tok (1.72x) |
| LUT | same 412 MB artifact | 7,830.8 MB (1.48x — worse than baseline) | 85.22 ms/tok (1.83x — slowest) |
| Combined RQ+NF4 | ~2.2 GB | 2,353.6 MB (0.45x — best of all six) | 80.12 ms/tok (1.72x) |
| MLP RQ v2 (this page) | ~1.66B params @ 9.44 bpp (~41% smaller on 63.6% of the model) | 5,279 MB (1.0x — no runtime savings) | 65.18 ms/tok (1.40x) |
The strongest result of the two new techniques on this page's family. MLP RQ v2 beats baseline on open-domain accuracy (+4.4), loses only 0.6 points on grounded accuracy, and matches exactly on grounding rate — the second-best accuracy profile of all six variants, on the single largest compression target in the whole project. The honest caveat: like embedding-only RQ, this is a storage win, not a runtime-memory win — the compressed matrices are reconstructed to full bf16 before serving, so peak GPU memory during generation is unchanged from baseline. A LUT-style non-materializing path for MLP weights would be a natural follow-up, not implemented here.
This is the least-validated technique in the project — MLP/attention weights are more precision-sensitive than an embedding table (they directly shape every attention and MLP computation, unlike a lookup-only embedding), and this is the first time independent per-matrix compression at this scale (78 matrices, no cross-layer sharing) has been tested end-to-end. The +15.3% relative val-loss increase and the accuracy numbers above are real, measured results — not a guarantee this generalizes beyond this project's held-out set.
Generation is capped at 80 tokens, same as every other Gemma-2-2B demo in this project.