Qwen3.5-4B-M4-ex-LRP
Ex-LRP (Explainable LRP) merge per the recipe in mergekit PR #682, using AttnLRP relevance scores (lxt) as the importance signal driving merge weighting. Backbone: mergekit master + the pr-682-exlrp branch.
This is the apples-to-mergekit-recipe variant. The same LRP signal driven through our dare_ties_merge.py OMv2 recipe is published as M5 — comparing M4 vs M5 isolates "merger" from "signal."
Sources
| Role | Model |
|---|---|
| Base | Qwen/Qwen3.5-4B |
| Source A | Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2 |
| Source B | Crownelius/Crow-4B-Opus-4.6-Distill-Heretic_Qwen3.5 |
Weights: 0.55 (A) / 0.45 (B). Density: 0.53. Seed: 42.
Phase 1+2 comparison (Q6_K)
| # | Recipe | Merger | Importance | HumanEval pass@1 | MBPP pass@1 |
|---|---|---|---|---|---|
| floor | Qwen/Qwen3.5-4B (no merge) | — | — | 60.37% 🥇 | 45.80% |
| src A | Jackrong-v2 (Claude-4.6-Opus reasoning distill) | — | — | 60.37% | 45.00% |
| src B | Crow-4B (Opus-4.6 distill, heretic) | — | — | 54.88% | 48.20% |
| M1 | Vanilla DARE-TIES | dare_ties_merge.py | none | 51.22% | 47.00% |
| M2 | OMv2 recipe (OBIM-lite + DAREx-q + EMR election) | dare_ties_merge.py | none | 52.44% | 49.40% |
| M3 | OMv2 + Fisher | dare_ties_merge.py | Fisher | 57.93% | 48.80% |
| M4 (this) | ex-LRP (mergekit PR #682) | mergekit (PR #682) | LRP | 51.22% | 49.40% |
| M4-v2 | ex-LRP (PR #682 turbo, w=1/1, d=0.7) | mergekit PR #682 turbo | LRP | 55.49% | 52.20% 🥇 |
| M5 | OMv2 + LRP | dare_ties_merge.py | LRP | 53.05% | 51.40% |
Eval methodology: llama-server (--reasoning-format deepseek --reasoning-budget 8192 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 -c 32768) → lm_eval local-completions against raw /v1/completions, temperature 0, max_gen_toks=2048. All five variants scored under identical conditions.
Floor reference: the top three rows are the base model and the two source fine-tunes evaluated under the same methodology. Hard finding: no merge beats Qwen3.5-4B base on HumanEval (best merge M3 = 57.93%, base = 60.37%). On MBPP, M4-v2 (52.20%) and M5 (51.40%) clearly beat both sources (best source: Crow at 48.20%) — merging does add MBPP capability, but borrows HumanEval ground.
LRP signal
Under lrp/ you'll find the AttnLRP relevance score safetensors used for this merge (multimodal-prefixed form: model.language_model.X), one per source model. Computed via scripts/lrp_compute_calibration.py on the same 64×256-token calibration mix as the Fisher run for M3.
The same raw scores in bare-keyed form (model.X) — what dare_ties_merge.py expects — live under lrp/ in the M5 repo. The two are losslessly interconvertible via scripts/rename_lrp_keys_for_multimodal.py.
Patches required for multimodal Qwen3.5_5
Running PR #682 against Qwen3_5ForConditionalGeneration required several minor patches (will be forwarded to the PR):
mergekit/architecture/base.py— pydantic forward-refmodel_rebuild()calls.mergekit/architecture/auto.py— layer-awareoptionalfor hybrid attention modules.mergekit/merge_methods/lrp.py— base-passthrough fallback for tensors without LRP scores (vision tower, MTP).mergekit/lrp_computer.py— supportqwen3_5_textinner-LM dispatch + tied-tensor clone before save.mergekit/config.py—strallowed inParameterSettingunion.lxt/efficient/models/__init__.py— optionalvit_torchimport.
Other variants in this study
- Qwen3.5-4B-M1-Dare-Ties — vanilla DARE-TIES
- Qwen3.5-4B-M2-OMv2 — OMv2 recipe (no importance signal)
- Qwen3.5-4B-M3-Fisher — OMv2 + Fisher
- Qwen3.5-4B-M4-ex-LRP — mergekit PR #682 ex-LRP
- Qwen3.5-4B-M4-v2-ex-LRP-turbo — mergekit PR #682 turbo retest
- Qwen3.5-4B-M5-OMv2-LRP — OMv2 + LRP