Qwen3.5-4B-M5-OMv2-LRP
OMv2 recipe (OBIM-lite + DAREx-q + EMR election) with AttnLRP relevance scores as the importance signal driving DAREx-q sparsification. Best MBPP result of the study (51.40%) and best balanced score (53.05% / 51.40%).
Apples-to-apples comparison against M3 (OMv2 + Fisher) — same merger, same recipe, only the importance source differs (LRP relevance vs Fisher squared-grad). And against M4 (mergekit's ex-LRP) — same LRP signal, different merger.
Sources
| Role | Model |
|---|---|
| Base | Qwen/Qwen3.5-4B |
| Source A | Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2 |
| Source B | Crownelius/Crow-4B-Opus-4.6-Distill-Heretic_Qwen3.5 |
Weights: 0.55 (A) / 0.45 (B). Density: 0.53. Seed: 42.
Phase 1+2 comparison (Q6_K)
| # | Recipe | Merger | Importance | HumanEval pass@1 | MBPP pass@1 |
|---|---|---|---|---|---|
| floor | Qwen/Qwen3.5-4B (no merge) | — | — | 60.37% 🥇 | 45.80% |
| src A | Jackrong-v2 (Claude-4.6-Opus reasoning distill) | — | — | 60.37% | 45.00% |
| src B | Crow-4B (Opus-4.6 distill, heretic) | — | — | 54.88% | 48.20% |
| M1 | Vanilla DARE-TIES | dare_ties_merge.py | none | 51.22% | 47.00% |
| M2 | OMv2 recipe (OBIM-lite + DAREx-q + EMR election) | dare_ties_merge.py | none | 52.44% | 49.40% |
| M3 | OMv2 + Fisher | dare_ties_merge.py | Fisher | 57.93% | 48.80% |
| M4 | ex-LRP (mergekit PR #682) | mergekit (PR #682) | LRP | 51.22% | 49.40% |
| M4-v2 | ex-LRP (PR #682 turbo, w=1/1, d=0.7) | mergekit PR #682 turbo | LRP | 55.49% | 52.20% 🥇 |
| M5 (this) | OMv2 + LRP | dare_ties_merge.py | LRP | 53.05% | 51.40% |
Eval methodology: llama-server (--reasoning-format deepseek --reasoning-budget 8192 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 -c 32768) → lm_eval local-completions against raw /v1/completions, temperature 0, max_gen_toks=2048. All five variants scored under identical conditions.
Floor reference: the top three rows are the base model and the two source fine-tunes evaluated under the same methodology. Hard finding: no merge beats Qwen3.5-4B base on HumanEval (best merge M3 = 57.93%, base = 60.37%). On MBPP, M4-v2 (52.20%) and M5 (51.40%) clearly beat both sources (best source: Crow at 48.20%) — merging does add MBPP capability, but borrows HumanEval ground.
LRP signal
Under lrp/ you'll find the AttnLRP relevance score safetensors used for this merge (bare-keyed form: model.X, what dare_ties_merge.py expects when the model is loaded as Qwen3_5ForConditionalGeneration and named_parameters() yields the inner-LM short paths). One file per source.
The same raw scores in multimodal-prefixed form (model.language_model.X) are published under lrp/ in the M4 repo. The two are losslessly interconvertible via scripts/rename_lrp_keys_for_multimodal.py.
Other variants in this study
- Qwen3.5-4B-M1-Dare-Ties — vanilla DARE-TIES
- Qwen3.5-4B-M2-OMv2 — OMv2 recipe (no importance signal)
- Qwen3.5-4B-M3-Fisher — OMv2 + Fisher
- Qwen3.5-4B-M4-ex-LRP — mergekit PR #682 ex-LRP
- Qwen3.5-4B-M4-v2-ex-LRP-turbo — mergekit PR #682 turbo retest
- Qwen3.5-4B-M5-OMv2-LRP — OMv2 + LRP