Qwen3.5-4B-M4-v2-ex-LRP-turbo

Re-merge of M4-ex-LRP using the updated mergekit PR #682 (Tusm11's feature/ex-lrp branch, HEAD 8d989f6 — "supercharged ex-LRP with turbo optimizations and multimodal support").

Same AttnLRP signal as M4-orig, same sources, but:

• Updated PR #682 code (Iron-Man stabilization, in-place math, multimodal support).
• Different merge hyperparameters: equal weights (1.0 / 1.0) and density 0.7 — closer to PR #682's worked example than the M4-orig 0.55/0.45/0.53 search.

This isolates "what changed when the merger code was updated AND hyperparams were rebalanced" — not a clean ablation of the code change alone, but the closest fair retest given the original LRP signal is reused.

Sources

Merge config:

merge_method: lrp
base_model:
  model: Qwen/Qwen3.5-4B
parameters:
  density: 0.7
models:
  - model: Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2
    parameters:
      weight: 1.0
      lrp_scores: jackrong-v2_lrp_scores.safetensors
  - model: Crownelius/Crow-4B-Opus-4.6-Distill-Heretic_Qwen3.5
    parameters:
      weight: 1.0
      lrp_scores: crow-4b_lrp_scores.safetensors

LRP scores reused from the original M4 run — multimodal-prefixed form (model.language_model.X), 427 keys, 19.4 GB each. Available under lrp/ in M4-ex-LRP.

Phase 1+2 comparison (Q6_K)

Eval methodology: llama-server (--reasoning-format deepseek --reasoning-budget 8192 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 -c 32768) → lm_eval local-completions against raw /v1/completions, temperature 0, max_gen_toks=2048. All variants scored under identical conditions on the same hardware (RTX 3090).

Delta vs M4-orig: HE +4.27 pp, MBPP +2.80 pp. The "turbo" branch + rebalanced hyperparams beat the original PR #682 run by a clear margin on both benchmarks. M4-v2 takes the MBPP lead of the whole study (52.20%) while staying competitive on HumanEval.

Patches required for multimodal Qwen3.5

Running PR #682 (turbo head) against Qwen3_5ForConditionalGeneration required the same set of patches as the M4-orig run, with a couple additions surfaced by the new code path:

• mergekit/architecture/base.py — import torch for pydantic forward-ref model_rebuild() (PretrainedConfig schema references torch.dtype).
• mergekit/architecture/auto.py — layer-aware optional for hybrid linear-attn / full-attn templates.
• mergekit/merge_methods/lrp.py — base-passthrough fallback for tensors without LRP scores (vision tower, MTP heads).
• mergekit/sparsify.py — added build_mask(importance, density) helper used by the turbo LRPMergeTask.
• mergekit/common.py — added .get() and __contains__ on ImmutableMap (turbo make_task calls params.get("weight", 1.0)).
• mergekit/config.py — str allowed in ParameterSetting union (so lrp_scores: "<path>" validates).

Floor reference: the top three rows are the base model and the two source fine-tunes evaluated under the same methodology. Hard finding: no merge beats Qwen3.5-4B base on HumanEval (best merge M3 = 57.93%, base = 60.37%). On MBPP, M4-v2 (52.20%) and M5 (51.40%) clearly beat both sources (best source: Crow at 48.20%) — merging does add MBPP capability, but borrows HumanEval ground.

Other variants in this study

• Qwen3.5-4B-M1-Dare-Ties — vanilla DARE-TIES
• Qwen3.5-4B-M2-OMv2 — OMv2 recipe (no importance signal)
• Qwen3.5-4B-M3-Fisher — OMv2 + Fisher
• Qwen3.5-4B-M4-ex-LRP — mergekit PR #682 ex-LRP (original)
• Qwen3.5-4B-M5-OMv2-LRP — OMv2 + LRP

License

Apache-2.0, inheriting from the base model.