Qwen3.5-4B-MicroCoder

A 4B-parameter code-leaning merge of Qwen3.5-4B that beats every individual source on LCB-medium-55, while preserving full GSM8K parity with the strongest reasoning fine-tune in the pool.

This card documents Qwen3.5-4B-MicroCoder (internally v2i-jv-base-task-arith), the chosen frontier point of a 19-variant ablation that swept merge methods, density, importance signals, AIME-protection masks, and skip-layer surgery.

Built with OmniMergeKit — the open-source merge engine developed for this work.

Headline numbers (Q6_K, greedy)

Net: +7pp MBPP, +3.3pp LCB-30, +1.8pp LCB-55, GSM8K parity. Trade-offs are HumanEval (−3pp), MMLU-Pro (−4.4pp), and the AIME math-reasoning floor (see "Why no AIME?" below).

Recipe

python omnimergekit.py \
    --base       Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2 \
    --task-base  Qwen/Qwen3.5-4B \
    --source     coder_eval/continuum-code-forged \
    --source     coder_eval/jackrong-python \
    --method omnimerge_v2 --v2-features fisher,darex \
    --weights 0.55,0.45 --density 0.53 --darex-q 0.85 \
    --fisher continuum-forged.safetensors,jackrong-python.safetensors \
    --pr682-turbo \
    --seed 42 --device cuda

This is a task-arithmetic merge:

MicroCoder = jackrong-v2 + 0.55·DARE(continuum-code-forged − base) + 0.45·DARE(jackrong-python − base)

• jackrong-v2 is the merge base — its full output style and reasoning policy survive intact at zero deltas. The two coding teachers contribute only their delta from the official Qwen3.5-4B base, not their absolute representations. This isolates "what the coder fine-tunes added on top of the public base" and grafts that onto the reasoning-distilled model.
• DAREx-q 0.85 drops the bottom 85% of cf/jp deltas by magnitude (per-tensor quantile) before random pruning, then rescales by 1/density. This kills low-magnitude noise while preserving the high-amplitude code-skill structure.
• Fisher importance from forward-pass gradient maps over the coder fine-tunes' own training-style data weights the EMR election so dominant per-element directions win when the two coding teachers disagree.
• PR682-turbo protects critical layers (norms, embeddings, lm_head, biases) at density 1.0 and falls back gracefully on shape mismatch.

Sources

Evaluation methodology

All evaluations: lm-evaluation-harness, llama.cpp llama-server with the published Q6_K quantization, /v1/completions raw endpoint, greedy decoding (temperature=0.0, top_p=1.0), max_gen_toks=2048 for HE/MBPP, max_gen_toks=8192 for LCB, --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0.

LiveCodeBench: medium-difficulty functional problems with min_date=2024-10-01 (post-Qwen3.5 training cutoff to avoid contamination). LCB-30 = first 30 problems of that pool, LCB-55 = full pool of 55.

Experiment trail (why this recipe?)

19 variants were ablated over a multi-week sweep. Summary table for the informative subset:

Key findings (apply to future merge work)

1. Task-arithmetic with the strong source as merge_base wins over symmetric DARE-TIES when one source is much stronger on the target axis (here: reasoning). v2g and v2i tie on LCB-55 (27.27%) but v2i wins HE/HE+/GSM8K and retains a small AIME signal that pure DARE-TIES kills.

2. Skip mlp.gate_proj layers 18-25 is the load-bearing AIME-recovery knob (+6.7pp). This maps from Qwen3.6's think-policy band 27-52/64 → 32-layer Qwen3.5 = 14-26 conservative narrow 18-25. Wider bands (v2k 14-27) collapse; full-MLP skip (v2l) destroys code skill.

3. DAREx-q 0.92 → 0.95 adds 3.3pp AIME on top of the skip by killing more low-magnitude cf/jp deltas in the protected reasoning band. 0.95 → 0.97 saturates (v2n=v2o on AIME).

4. The jv-AIME fisher suppression mask is fully redundant with skip-layers (v2n=v2p=v2q at AIME 13.33 and code metrics within noise). Per-element scaling cannot substitute for layer-level passthrough — jv's reasoning lives in the coherent per-layer behavior of mlp.gate_proj 18-25, not in the highest-importance individual cells. Mask alone (v2r) gives nothing.

5. The 13.33% AIME ceiling is structural, not a tuning problem. Three different mechanisms (high darex, higher darex, mask) all converge at the same number. Closing the remaining 13.34pp gap to jv source requires SFT distillation, not more merge tuning.

Why no AIME on the chosen variant?

MicroCoder (v2i) is the code-leaning frontier point. The skip-layer recipe (v2n) recovers AIME to 13.33% but at a 6.7pp LCB-30 regression. v2i preserves the better LCB; the trade is real and structural. A reasoning-leaning sibling exists internally (v2n) but is not published — LCB regression makes it strictly worse than jackrong-v2 for math users who already have access to the original.

Files

• Full-precision safetensors weights (BF16). Use ManniX-ITA/Qwen3.5-4B-MicroCoder-GGUF for the Q6_K quantization.

Use

llama-server -m Qwen3.5-4B-MicroCoder-Q6_K.gguf \
    --port 8099 -c 32768 -t 12 -ngl 99 \
    --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0

Greedy temperature=0.0, top_p=1.0 recommended for code tasks.

Citation

If you use this model or the OmniMergeKit recipes in your work:

@misc{mannix2026microcoder,
  title  = {Qwen3.5-4B-MicroCoder: a task-arithmetic merge for code},
  author = {Mannix, F.},
  year   = {2026},
  url    = {https://huggingface.co/ManniX-ITA/Qwen3.5-4B-MicroCoder},
  note   = {Built with OmniMergeKit, https://github.com/mann1x/omnimergekit}
}

License

Apache 2.0, inherited from Qwen3.5-4B and the source fine-tunes.