Qwen3.5-4B-M3-Fisher
OMv2 recipe (OBIM-lite + DAREx-q + EMR election) with diagonal Fisher information weighting as the importance signal driving DAREx-q sparsification. Best HumanEval result of the study (57.93%, +5.49 pp over recipe alone).
Note: An earlier version of this checkpoint silently dropped the Fisher signal due to a tensor-name prefix mismatch (Fisher files keyed by
model.Xfromnamed_parameters(), but Qwen3.5_5's safetensors index usesmodel.language_model.X). This release uses a prefix-aware Fisher lookup; Fisher signal is now actually applied. The buggy variant was bit-identical to M2 — see comparison table below for the correction.
Sources
| Role | Model |
|---|---|
| Base | Qwen/Qwen3.5-4B |
| Source A | Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2 |
| Source B | Crownelius/Crow-4B-Opus-4.6-Distill-Heretic_Qwen3.5 |
Weights: 0.55 (A) / 0.45 (B). Density: 0.53. Seed: 42.
Fisher: 64×256-token fp32 calibration each source (see fisher/ subdir for the actual safetensors and methodology).
Phase 1+2 comparison (Q6_K)
| # | Recipe | Merger | Importance | HumanEval pass@1 | MBPP pass@1 |
|---|---|---|---|---|---|
| floor | Qwen/Qwen3.5-4B (no merge) | — | — | 60.37% 🥇 | 45.80% |
| src A | Jackrong-v2 (Claude-4.6-Opus reasoning distill) | — | — | 60.37% | 45.00% |
| src B | Crow-4B (Opus-4.6 distill, heretic) | — | — | 54.88% | 48.20% |
| M1 | Vanilla DARE-TIES | dare_ties_merge.py | none | 51.22% | 47.00% |
| M2 | OMv2 recipe (OBIM-lite + DAREx-q + EMR election) | dare_ties_merge.py | none | 52.44% | 49.40% |
| M3 (this) | OMv2 + Fisher | dare_ties_merge.py | Fisher | 57.93% | 48.80% |
| M4 | ex-LRP (mergekit PR #682) | mergekit (PR #682) | LRP | 51.22% | 49.40% |
| M4-v2 | ex-LRP (PR #682 turbo, w=1/1, d=0.7) | mergekit PR #682 turbo | LRP | 55.49% | 52.20% 🥇 |
| M5 | OMv2 + LRP | dare_ties_merge.py | LRP | 53.05% | 51.40% |
Eval methodology: llama-server (--reasoning-format deepseek --reasoning-budget 8192 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 -c 32768) → lm_eval local-completions against raw /v1/completions, temperature 0, max_gen_toks=2048. All five variants scored under identical conditions.
Floor reference: the top three rows are the base model and the two source fine-tunes evaluated under the same methodology. Hard finding: no merge beats Qwen3.5-4B base on HumanEval (best merge M3 = 57.93%, base = 60.37%). On MBPP, M4-v2 (52.20%) and M5 (51.40%) clearly beat both sources (best source: Crow at 48.20%) — merging does add MBPP capability, but borrows HumanEval ground.
Other variants in this study
- Qwen3.5-4B-M1-Dare-Ties — vanilla DARE-TIES
- Qwen3.5-4B-M2-OMv2 — OMv2 recipe (no importance signal)
- Qwen3.5-4B-M3-Fisher — OMv2 + Fisher
- Qwen3.5-4B-M4-ex-LRP — mergekit PR #682 ex-LRP
- Qwen3.5-4B-M4-v2-ex-LRP-turbo — mergekit PR #682 turbo retest
- Qwen3.5-4B-M5-OMv2-LRP — OMv2 + LRP