Gemma 4 A4B 98-Expert v6-coder (C6v3lcb) — LCB-targeted code prune (~20.8B)

✅ Chat template — agentic loop fix (2026-06-10)

The chat template inherited from Gemma 4 upstream re-injected each prior tool-call turn's reasoning_content as a <|channel>thought block (for assistant tool-call steps after the last user message). In deep multi-turn agentic / tool-calling use (OpenCode, llama.cpp --jinja, …) this fed the model an accumulating echo of its own reasoning and could trigger verbatim repetition loops. Single-turn chat is unaffected.

Validated on Q6_K: agentic loop rate 33% → 0%, with HumanEval+ 92.07% and MultiPL-E-100 0.66 unchanged (the fix is single-turn-inert). A drop-in chat_template.fixed.jinja and a 1-minute template_loop_unittest.py are provided for audit. The bug is upstream-wide across the Gemma 4 family + third-party requants — see the writeup.

As of 2026-06-10 the corrected template is applied to chat_template.jinja in this repo.

Eval complete (Q6_K / llama.cpp). The full canonical 9-bench suite plus the extended LCB-medium-100 and MultiPL-E-100 code benches are filled below, every cell read from summary.json (greedy, cohort-pinned recipe). The 128e and 98e-v5-coder anchor columns are the matching Q6_K reference runs. NVFP4A16 is a deployment format and is not separately benchmarked (cohort policy).

Headline — the LCB targeting worked. LCB-medium-55 96.36% (+10.91pp vs v5-coder, and +1.81pp past the unpruned 128e), closing the −9.10pp hole that motivated the recipe; MultiPL-E macro 88.0% (+7 vs v5-coder, ≈128e); AIME recovers +10pp (53.33 → 63.33). The budget is paid on the non-code axes the LCB-only class weights deprioritized (MATH −3, IFEval −2 vs v5-coder).

A research checkpoint that prunes the unpruned Gemma 4 26B-A4B-it (128 experts, top-8 + shared, 30 layers) down to 98 experts per layer using a drop map that is the most code-faithful member of the v6-coder family: v5-coder's gentle C6 layer-relevance-weighted v4-floor recipe, re-derived on the corrected v3 code-pass calibration data, then steered specifically at LiveCodeBench-medium — the one code bench where expert pruning hurt most (−9.10pp vs 128e on v5-coder). Same 98e shape, same router, same attention, same norms as the rest of the cohort, plus the mandatory shared-FFN α=1.2 upweight all coder variants carry.

This is the head of the C6 → C6v3 → C6v3lcb line: each step holds the gentle C6 recipe fixed and changes exactly one variable (data, then target weighting), so the deltas are attributable.

Quantized formats

Eval is llama.cpp / Q6_K only for this cohort. NVFP4A16 is published as a vLLM-deployable format but is not benchmarked separately — the Q6_K scoreboard below is representative of the model's quality.

At a glance

---

Recipe

The drop map is produced by generate_drop_map_multiclass.py (omnimergekit) from per-expert, per-class contribution scores, then applied with expert_drop.py, then the shared expert is upweighted. Four design choices define C6v3lcb; each is held constant from v5-coder except where noted.

1. C6 gentle base recipe (unchanged from v5-coder)

The aggregation that ranks experts for dropping:

strategy      = max              # per-expert score = MAX over classes (not mean/geomean)
normalize     = rank             # rank-normalize within each (layer, class) before aggregating
protect_top   = 16               # the 16 highest-scoring experts/layer are never dropped
alpha         = 2.0              # contribution sharpening exponent
breadth_bonus = 0.5              # reward experts active across many classes (anti-overfit)
v4_floor_map  = v4_layer_floor_map.json   # per-layer floor: never drop below v4's keep on protected layers
baseline      = teacher_force_98e_p16_clean.json   # tie-break anchor

strategy=max + breadth_bonus is the load-bearing pair: it favors experts that are strongly useful to at least one class and broadly useful across classes, rather than experts with a high average that no single task depends on. This is the optimizer-off-manifold lesson encoded as a recipe — max/percentile beats mean/geomean for importance.

2. v3 calibration data + outlier fix (changed from v5-coder)

v5-coder ranked experts using the _fixed code-pass traces, which turned out to be ~86% NaN in the deep layers — accidentally NaN-blind where the model has real signal. v6-coder uses the corrected expert_neuron_v5_code_v3.json (real deep-layer signal restored, T73.0 fp32-hot-path patch), and scrubs the residual bf16 weight-norm artifacts with a median-based outlier clamp:

data                = expert_neuron_v5_code_v3.json
outlier_mode        = median
outlier_wnorm_thresh = 1e4       # clamp expert weight-norms above 1e4 to the layer median

Calibration corpus (v5_code_pass_traces.json, 360 traces): Tier-A 128-token comprehension prompts (342 traces, 1×) + Tier-B 2048-token windowed pass-traces (18 traces, 3×) — the long-window tier captures sustained code reasoning, which is where pruned variants ruminate.

3. LCB-only targeting — the "lcb" in C6v3lcb (changed)

Eight contribution classes are scored; the class weights steer which specialists are protected. v6-coder zeroes the HumanEval targeting and concentrates the targeted budget on LiveCodeBench-medium:

Rationale: HE/HE+ were already at/above 128e on v5-coder (+1.22 / +1.22), so spending protection budget on them is wasted; LCB-medium was the −9.10pp hole. Removing the HE targeting frees the floor to protect LCB-relevant experts harder. The resulting map is 95% identical to C6v3 (generic_code 3× dominates), so HE+/IFEval are expected at ≈C6v3 — the bench that tests the hypothesis is LCB-medium.

4. Mandatory shared-FFN α=1.2 (cohort rule)

After expert drop, router_shared_upweight.py --alpha 1.2 --target mlp.down_proj.weight scales the always-on shared expert's down-projection by 1.2×. Without it the pruned model is the "weak/ruminating" pre-T18 baseline; every coder variant carries it or the v5-coder comparison is unfair. Verified by the .shared_applied marker in this repo.

---

Expert mapping

• Uniform 30 experts dropped per layer, all 30 layers → 98 kept/layer (128 → 98).
• Mean overlap with the teacher-force baseline: 17.8 / 30 dropped per layer (min 11, max 24). The recipe re-ranks ~12 of 30 dropped slots/layer away from the naive teacher-force drop — that re-ranking is where the code-faithfulness lives.
• Aggregated keep scores (rank-normalized, α=2.0, breadth-bonus) span ~0.0–3.5 per layer with mean ~1.85; the protected top-16 sit at the high end, the dropped 30 at the low end, with boundary_ties_within_1pct 2–39/layer (highest in the shallow layers 0/10 — those layers have the flattest expert-importance profile).

Full per-layer keep lists are in expert_drop_metadata.json (per_layer_keep) in this repo; the ranking provenance (per-layer agg min/max/mean, ties, overlap-vs-baseline, v5-only non-overlap experts) is in scripts/v6coder_C6v3lcb_drop_map.json.summary.json.

---

Problems solved

1. Rumination on hard code/reasoning problems. Pruned Gemma 4 variants fall into <think> loops on the hardest LCB / GPQA / AIME problems, blowing the token budget without converging. C6v3lcb attacks this two ways: (a) the LCB-targeted drop map keeps the experts that the hard-problem pass-traces actually use; (b) at eval time the lcb_medium_*_v4 template applies a thinking_token_budget=12288 forcing function (parser=gemma4 + enable_thinking) that caps the rumination loop. T109 rumination-signal gate (GPQA-48 / AIME probes) confirmed the variant generates and converges rather than looping.
2. Data-vs-recipe disentanglement (T102). v6-coder isolates whether v3's earlier regressions were the data or the recipe: holding v5-coder's gentle C6 recipe fixed and only swapping in v3 data + the outlier fix. C6v3 ≈ v5-coder in smoke → the v3 data is fine and C12's aggressive recipe was the rumination cause; C6v3lcb then re-steers that clean baseline at LCB.
3. Deep-layer calibration corruption (T73.0). The _fixed traces were ~86% NaN in deep layers; v6-coder uses real-signal v3 data with a median outlier clamp so deep-layer experts are ranked on genuine contribution, not NaN-blindness.
4. Unfair-comparison trap. The shared α=1.2 step is mandatory and marker-verified, so v6-coder vs v5-coder is apples-to-apples.

---

Scoreboard — Q6_K GGUF, llama.cpp, greedy

Full 9-bench llama.cpp Q6_K run, llama-server --reasoning-format deepseek --reasoning-budget 12288 --parallel 2, greedy (T=0, top_p=1, top_k=0, do_sample=false). The 128e and v5-coder columns are the bartowski-Q6_K / v5-coder-Q6_K reference runs under the identical recipe (apples-to-apples within the llama.cpp/Q6_K backend).

Read: v6-coder lands on/above v5-coder on 8 of 9 benches and beats the unpruned 128e on every code bench (HE +2.44, HE+ +2.44, LCB-55 +1.81). The headline is LCB-medium-55 +10.91pp vs v5-coder — the targeted hole is not just closed but pushed past the base model. AIME recovers +10pp (53.33 → 63.33). The cost lands on MATH (−3) and IFEval (−2) vs v5-coder — the non-code generalist axes the LCB-only class weights (1 1 3 1 1 0 0 2) deliberately deprioritized.

Extended code benches — LCB-medium-100 + MultiPL-E-100 (Q6_K, llama.cpp, greedy)

Run on solidpc (T112), same Q6_K / greedy recipe; MultiPL-E scored via the nuprl/multipl-e-evaluation Docker image. 128e and v5-coder columns from the v5-coder card.

LCB-medium-100 (100-problem superset of LCB-medium-55 v4):

(v6-coder also clears 128e on LCB-100 by +1.00pp — the LCB-targeting win holds on the 100-problem superset, not just the 55-problem v4 set.)

MultiPL-E-100 (HumanEval → Rust / Java / JS, 100/lang, chat-mode + code extraction):

v6-coder near-fully recovers MultiPL-E to the 128e level (macro −1.67pp) from v5-coder's −8.67pp gap — code generalization in non-Python languages tracks the LCB-targeting win, not just the in-distribution LCB benches. (264/300 passed; micro = macro = 88.0%.)

Coder-field comparison — v6-coder vs Qwen2.5-Coder-14B / 7B + Qwen3.5-9B (Q6_K, llama.cpp, greedy)

The 9 canonical benches + MultiPL-E-100, all on the identical llama.cpp Q6_K / greedy recipe (reasoning models served with --reasoning-format deepseek --reasoning-budget 12288 --parallel 2). Architectures differ — this is a same-harness comparison, not a same-class one:

• v6-coder — Gemma-4 26B-A4B MoE pruned to 98 experts (~20.8B total, ~A4B active), reasoning.
• Qwen2.5-Coder-14B / 7B-Instruct — dense, non-reasoning code specialists (bartowski Q6_K).
• Qwen3.5-9B — dense reasoning model (bartowski Q6_K).

† Qwen2.5-Coder-14B HumanEval+ / LCB-medium-55 reused from the same-stack GGUF HE+ sweep (not re-run in this chain). All comparison cells filled 2026-05-28 (9B Fix-A re-eval + 14B Q6_K reference). running legend: eval in flight — table updated as results land.

Note on Qwen3.5-9B. Qwen3.5-9B is a verbose, slow thinking model: it emits long <think> reasoning chains (often ≥1900 tokens even on a trivial GSM8K question), so it runs several× slower per question than the non-reasoning Qwen2.5-Coder models — well beyond what its 9B size would suggest. Its GSM8K / MATH-500 / GPQA cells are being re-run after a harness fix: under batched, reasoning-parsed serving the verbose thinking intermittently left the final answer inside the reasoning block (mis-scored as empty content). ARC (96.76%) and AIME (56.67%) were unaffected.

---

Answer-length analysis (anti-rumination)

v6-coder thinks longer on average than 128e on the code benches — it spends the full bounded thinking_token_budget=12288 where 128e often answers in a few hundred tokens. The question this section answers is whether that extra length is rumination (long thinking that fails to reach an answer — the failure mode the LCB targeting was built to fix) or productive reasoning that lands on a PASS. The tables below compare per-problem completion length against 128e and v5-coder on the same problems, from omk_eval token_stats (characters from the raw completion text; tokens via the 128e tokenizer), on the real-n benches. 128e and v5-coder are the matching Q6_K runs. The short answer: by the rumination-as-wasted-thinking definition, v6-coder ruminates less than 128e, not more — 128e's longest LCB outputs (up to 51k chars) are failures, while v6-coder's long outputs overwhelmingly pass.

Per-problem completion length — characters (p50 / p90 / max):

Per-problem completion length — tokens (p50 / p90 / max):

Budget-saturation incidence — share of problems whose completion reached ≥12k tokens (at/near the thinking_token_budget=12288 cap). Saturation by itself is not rumination: a saturated output that reaches a correct PASS is productive use of the budget. The pruned variants saturate on nearly everything; 128e almost never does — but see the next table.

Rumination — long thinking that fails to PASS. The right metric is not median length (128e looks short only because it answers easy problems fast). It is the share of the model's budget-saturated outputs that still fail — i.e. tokens burned without a correct answer:

Every 128e output that reaches the budget cap is a failure (100%), and 128e's failed problems run 5.7× longer than its passed ones (mean 8368 vs 1468 tok): 128e only thinks long when it is lost. v6-coder thinks long on ~97% of problems but 96% of those long outputs PASS — its long thinking is overwhelmingly productive, not rumination.

Key findings:

• LCB — v6-coder ruminates less than 128e, under the only definition that matters (long thinking that fails to reach a PASS). The naive read of the median says the opposite — 128e's median LCB completion is ~0.6–1.2k tokens vs v6-coder's budget-capped ~12.7k — but that short median is an artifact of mixing easy and hard problems. 128e only thinks long when it is lost, and then it fails: every 128e output that saturates the budget is a failure (2/2 on LCB-100, 1/1 on LCB-55), and the single longest output across all three models — 51,568 chars / 15,585 tokens on lcb/leetcode/3659 — is a 128e failure, one that both v6-coder (PASS, 12,691 tok / 32,676 chars) and v5-coder solve in two-thirds the length. v6-coder, by contrast, saturates the budget on ~97% of problems but 96% of those long outputs PASS; its longest LCB-55 output is a correct answer and its max length (41–43k chars) sits below 128e's 51k. Wasted long-thinking rate: 128e 100%, v6-coder ~4%. And v6-coder beats 128e on score on both axes (96.36 / 96.00 vs 94.55 / 95.00) — more correct and a lower worst-case tail.
• 128e's routing is erratic in both directions, and the targeted map repairs both. 128e fails LCB two ways: by over-thinking (3659: 51k chars, no answer) and by under-thinking (lcb/leetcode/3000: a premature wrong answer in 579 tokens; 3793: 3101 tok) — on the very same problems v6-coder spends the full ~13k-token budget and passes. This bidirectional instability is the fingerprint of an imperfect base router; the 51k outliers and the lower 128e score are the symptom. v6-coder's LCB-targeted expert map corrects 3 of the 5 problems 128e fails on LCB-100 (2 of 3 on LCB-55). The thinking_token_budget=12288 forcing-function only bounds the loop; the targeted map is what makes the bounded thinking land on a PASS instead of a 51k-char dead end. (Versus v5-coder, v6-coder is statistically the same length — median 12799 vs 12804 tok on LCB-55 — yet passes far more: 53/55 vs 47/55, 96/100 vs 91/100.)
• AIME — genuinely less rumination, +10pp. Median completion 939 tok vs v5-coder's 1156, p90 char tail 8404 < 9239, scoring 63.33% vs 53.33% — shorter chains and more correct.
• GSM8K — tightest of the three (max 396 tok; 128e 1107, v5-coder 858); no rumination tail.
• IFEval — no ruminating outlier (max 7200 chars vs v5-coder's 17539; ≈128e 7425).
• GPQA — a base-model ceiling, not a prune artifact. p90 completion tokens peg at ~8189 on all three models; the hard-question rumination is inherited from Gemma 4 and is identical across the cohort, not widened by pruning.
• ARC — one shared single-sample outlier (~16266 tok / ~44k chars) on both pruned variants, absent on 128e; p50/p90 normal and identical, so not systemic length growth.
• MultiPL-E — most compact code of the three. On the bench where the answer is the code (extracted final block, no reasoning trace), v6-coder has the tightest tail — max 2169 chars vs 128e 2944 / v5-coder 3083, p90 591 chars / 184 tok — at ≈128e pass-rate (macro 88.0%).
• Net: v6-coder is more accurate than 128e on every LCB axis (96.36 / 96.00 / MultiPL-E macro 88.0) at a higher average inference cost (it thinks the full bounded budget) but a lower worst case (max 41–43k chars vs 128e's 51k) and a far lower wasted-thinking rate (~4% vs 100%). On the short-reasoning benches (AIME / GSM8K / IFEval) it additionally reduces length vs v5-coder. The targeted map never traded length for accuracy — it traded 128e's erratic, sometimes-catastrophic tail for a higher but bounded and productive average.

Methodology / data integrity. Per-problem lengths come from omk_eval token_stats over each bench's per-problem samples archive — lm-eval samples_*.jsonl for the 9-bench, and the native runners' lcb_result.samples.jsonl / mpe_result.samples.jsonl (mirrored into the durable sqlite resume cache) for LCB/MPE. MultiPL-E is tabulated, but its rows measure extracted-code length, not reasoning: its samples store only the final code block (no <think> trace, finish_reasons all terminal, thinking_tokens_est=0), so MPE is a code-conciseness reference — not a rumination metric — on which v6-coder is the most compact. The reasoning-bearing benches (LCB / GPQA / AIME / MATH) carry the rumination signal; MPE tokens there are tokenizer-estimated (the code-extraction path returns no server token count). thinking_tokens_est is 0 across all benches under llama.cpp --reasoning-format deepseek (reasoning is emitted inline, not as a separable block), so the completion-token columns are the length signal. The LCB/MPE token_stats are computed over per-problem rows keyed on doc_id (the native runners now emit it, fixed 2026-05-25) — earlier card revisions briefly collapsed these to n=1; all numbers here are at full n (LCB-55 n=55, LCB-100 n=100, MultiPL-E n=300).

---

Reasoning budget and thinking stop phrase (llama.cpp)

On a hard prompt this model will reason until it has consumed the whole context window and then answer with nothing at all. llama.cpp can bound the thinking block with a sampler, and — the part that actually matters — tell the model why the block is being closed.

Needs llama.cpp b8508 or newer for the flags, b10091 or newer for the per-request overrides.

Serve with a bounded thinking block

llama-server -m gemma-4-A4B-98e-v6-coder-it-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-budget 8192 \
    --reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n' \
    --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.05 \
    --repeat-penalty 1.02 --repeat-last-n 2048

Both flags also read from the environment: LLAMA_ARG_THINK_BUDGET and LLAMA_ARG_THINK_BUDGET_MESSAGE.

--reasoning-format is not part of this. It only decides how the thinking is handed back — message.reasoning_content versus left inline in message.content — and never whether the budget is enforced: the delimiters the sampler counts are set by the chat template regardless, so the cap binds under auto, deepseek and none alike. The default auto already extracts reasoning and is behaviourally identical to deepseek (they differ only in name; the sole branch in the parser is != none). Leave it at the default so the model's own tool-call and channel handling stays in play, and pin deepseek only when a harness needs the thinking kept out of content.

--reasoning-budget on its own forces the closing tag the moment the budget runs out, wherever the model happens to be. When that lands mid-thought the model frequently does not register that it was interrupted: it carries on reasoning, now inside the visible answer. The stop phrase is what prevents that — it gives the model a reason to be finishing.

Two wordings that work

# "qwen" — the string Qwen's own service uses, from their docs
--reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n'

# "voice" — shorter, in the model's own reasoning voice
--reasoning-budget-message $'\n\nOK, I have enough to answer now.\n'

Wording is model-specific: Qwen note that the ability to act on such a message "is not explicitly trained but emerges naturally", so it is worth trying both on your own workload. Leading and trailing newlines matter — they keep the phrase off whatever half-finished line the cut landed on.

What it measures out to

Measured on the v7-coder IQ4_NL build of this family, served by the same llama.cpp sampler. Three hard questions, temperature 0.6, fixed seed, answer characters with wall time in brackets. Every run answered all three correctly, and thinking length is unchanged by the message in every row:

AIME 2024, all 30 problems, budget 4096, -c 32768, vendor sampling:

The phrase halves wall time and all but removes the runaway answers — single rows go from 82,067 characters of answer to 1,655. The accuracy differences are inside the noise at n = 30 (paired: qwen −4 net, voice −1 net, exact binomial p ≈ 0.22 and ≈ 1.0), and the terse "Final Answer:" suffix from the s1 paper (arXiv:2501.19393) is not reproducing the accuracy collapse reported there at this budget.

Per request, instead of per server

The server accepts both as request fields, overriding the command line:

{
  "messages": [ ... ],
  "thinking_budget_tokens": 8192,
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On the raw /completion endpoint the delimiters are not inferred, so they have to be supplied with the budget:

{
  "prompt": "...",
  "reasoning_budget_tokens": 8192,
  "reasoning_budget_start_tag": "<|channel>",
  "reasoning_budget_end_tag": "<channel|>",
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On b10091 the message field must be present on /completion requests even when empty: llama.cpp builds the sequence it forces from message + end_tag inside that field's handler, so omitting it leaves the budget with nothing to force — the sampler logs as though the cap fired while the thinking block stays open.

Rules of thumb

• Keep -c several times larger than the budget. A budget equal to the context lets the thinking phase fill the window on its own.
• A quarter of the context is a sensible starting point: 8192 at -c 32768.
• The budget is per thinking block, not per response — the sampler re-arms when it sees a new opening tag, so a multi-turn agent gets a fresh window each time.

Reproduce

# 1. drop map (omnimergekit)
python scripts/generate_drop_map_multiclass.py \
    --data scripts/expert_neuron_v5_code_v3.json --target 98 \
    --protect-top 16 --alpha 2.0 --strategy max --normalize rank \
    --breadth-bonus 0.5 --v4-floor-map scripts/v4_layer_floor_map.json \
    --baseline-drop-map scripts/teacher_force_98e_p16_clean.json \
    --outlier-mode median --outlier-wnorm-thresh 1e4 \
    --class-weights 1 1 3 1 1 0 0 2 \
    --output scripts/v6coder_C6v3lcb_drop_map.json

# 2. prune + mandatory shared upweight (see scripts/build_v6coder_C6v3lcb.sh)
python scripts/expert_drop.py --source-dir google/gemma-4-26B-A4B-it \
    --drop-map scripts/v6coder_C6v3lcb_drop_map.json --suffix=-v6-coder-C6v3lcb-it
python scripts/router_shared_upweight.py \
    --model-dir google/gemma-4-A4B-98e-v6-coder-C6v3lcb-it \
    --alpha 1.2 --target mlp.down_proj.weight

Lineage & provenance

• Base: google/gemma-4-26B-A4B-it (128e, fresh prune — not derived from v4/v5 weights).
• Cohort: v5-coder (C6 _fixed) → C6v3 (C6 v3-data) → C6v3lcb (this, LCB-targeted).
• Tooling: omnimergekit — expert_drop.py, generate_drop_map_multiclass.py, router_shared_upweight.py; eval via omk_eval.py + EVAL_PROTOCOL.
• Eval sampler is frozen GREEDY across the whole cohort for apples-to-apples comparison.

License for model weights:

• Gemma License (inherits from Google Gemma 4)

Additional files authored by ManniX-ITA:

• Apache-2.0

License: Gemma Terms of Use. This is a derivative of Gemma 4 and inherits those terms.