Huihui-gemma-4-26B-A4B-it-qat-abliterated-pad768-NVFP4

推奨 / Recommended: the MTP bundle → Huihui-gemma-4-26B-A4B-it-qat-abliterated-MTP-NVFP4 — same body + the gemma4_mtp assistant included in assistant/, one download for spec-decode (JA 134 / EN 163 tok/s vs 108 baseline).

NVFP4 (full W4A4) quantization of huihui-ai/Huihui-gemma-4-26B-A4B-it-qat-q4_0-unquantized-abliterated — the abliterated, QAT-q4_0-origin Gemma 4 26B-A4B instruct MoE (128 experts, top-8), with the MoE intermediate zero-padded 704 → 768 so it loads on a stock vLLM (CUTLASS NVFP4 MoE kernel needs /128 alignment).

Lineage: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized (QAT q4_0 → bf16) → huihui-ai abliteration → NVFP4 (W4A4) → loss-less 704→768 pad (this repo).

51.6 GB → 17.6 GB. Serves on 2× 16 GB Blackwell GPUs (TP=2) at ~107 tok/s single-stream.

The finding: QAT fixes the MoE W4A4 Japanese long-form collapse

The non-QAT build of this exact architecture (Huihui-gemma-4-26B-A4B-it-abliterated-pad768-NVFP4) has a known failure mode: Japanese long-form generation intermittently collapses into repetition loops (get(get)get(get)…) past ~500 tokens, while English code/math stays fine. The working theory was FP4 quantization noise breaking the expert routing combinations that non-English text needs.

This QAT-origin build does not collapse. Two adversarial long-Japanese probes (the same Hōjōki-essay class that broke the non-QAT model, plus a four-seasons essay pushed against a 2048-token cap):

• 1360 tok and 1346 tok completions, both ending in a natural EOS (finish_reason: stop)
• Zero hits from a mechanical loop detector (any 2–30-char block repeated ≥6× consecutively)
• Coherent, on-topic, stylistically controlled Japanese prose to the final sentence

Combined with the dense-31B QAT result (same recipe, same survival), the pattern now holds across dense and MoE: if you want gemma-4 in NVFP4 W4A4, go through a QAT checkpoint — the q4_0-shaped weight distribution is the prior that FP4 wants, and on MoE it is the difference between collapse and sanity. (Caveat for strict A/B readers: the non-QAT bake used a different calibration set — neuralmagic/calibration 20×8192 vs ultrachat_200k 256×2048 here — so calibration is not held perfectly constant, but the 31B/12B dense evidence points squarely at QAT.)

English is unaffected: clean iterative fib(n) with correct asserts and complexity notes (HumanEval-class behavior matches the non-QAT sibling, which scored 90.8% HumanEval+).

Quality evidence (Japanese, temp 0.7 — verbatim)

• 「一文で自己紹介して。」→ 「私はGoogle DeepMindによって開発された、テキストと画像を理解し、テキストで回答を生成することができるオープンウェイトの大型言語モデル、Gemma 4です。」
• 太郎>花子>次郎 reasoning → 「一番背が低いのは次郎です。… 太郎 ＞ 花子 ＞ 次郎 という順序（身長が高い順）になるため、一番低いのは次郎となります。」 (correct, clean chain)
• 春の俳句 → 「日向ぼこ　猫も心も　春隣」 (5-7-5 with spring kigo, plus self-commentary)

Observed imperfection worth disclosing: at temp 0.7 we saw two isolated single-token glitches across all probes (「次_郎」「社会情str」) — instantly self-recovered, no loops, no degeneration. The non-QAT failure mode (sustained collapse) never appeared.

Serving with vLLM

Requires a Blackwell GPU (SM120) and vLLM ≥ 0.21 (compressed-tensors NVFP4 auto-detect — no --quantization flag). Weights are ~16.4 GiB — they do not fit a single 16 GB card; use 2× 16 GB (TP=2) or a single ≥ 24 GB GPU.

TP=2 (2× 16 GB) — the sweet spot

NCCL_P2P_DISABLE=1 vllm serve sakamakismile/Huihui-gemma-4-26B-A4B-it-qat-abliterated-pad768-NVFP4 \
  --served-model-name gemma4-26b-qat \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --kv-cache-dtype fp8 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 8192 \
  --limit-mm-per-prompt '{"image":0}'

• NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce are required on PCIe no-NVLink boxes (TP hangs without them); drop both if you have NVLink/P2P.
• Keep CUDA graphs ON (no --enforce-eager).
• --kv-cache-dtype fp8 doubles KV capacity; '{"image":0}' serves text-only and skips multimodal profiling.

Measured (RTX PRO 2000 Blackwell 16 GB, PCIe no-NVLink, CUDA graphs, fp8 KV)

Same speed class as the non-QAT sibling (slightly faster here) — the QAT provenance costs nothing at inference and buys back Japanese long-form.

Measured (TP=2 vs TP=4) — this QAT build, same box, same flags

TP=2 is the sweet spot. Doubling GPUs buys only +6% single-stream / +11–14% aggregate — per-GPU throughput (and therefore tok/joule) nearly halves at TP=4. The MoE rule observed on the non-QAT sibling holds here, and is even stronger: the non-QAT build gained +28% single-stream going to TP=4, this QAT build only +6%. Quality at TP=4 is identical (same sane self-introduction, natural EOS). Spend the extra GPUs on a second TP=2 replica instead.

Speculative Decoding (measured 2026-06-12)

Three drafting strategies measured single-stream (T=0, chat completions, ×3 each) against this NVFP4 body on TP=2 GPU0,1, fp8 KV, vLLM 0.21.0. Acceptance = accepted/drafted tokens from /metrics diff.

Winner: native MTP — google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant (bf16, 832 MB), method auto-normalized to gemma4_mtp:

--speculative-config '{"method":"gemma4_mtp","model":"<drafts>/google-26b-mtp-assistant","num_speculative_tokens":4}'
# resident: maxlen 16384 / GMU 0.92 / MBT 8192 / fp8 KV → KV 23,560 tok (assistant costs ~0.4 GB/GPU)

• Concurrent (MTP, aggregate): 4 / 8 streams × 256 tok (×3 avg, diverse prompts): MTP 341.5 / 557.6 tok/s JA (363.4 / 566.4 EN) vs baseline 326.3 / 571.1 → MTP pays up to ~4 streams (+4.7% JA), break-even at 8 (−2.4%, run noise). Acceptance stays 36–47% under batch; the gain shrinks because the MoE saturates compute at 8 streams, not because drafting degrades. Keep MTP resident: peak throughput unharmed, single-stream 1.2–1.5×.
• Japanese caveat: the EAGLE-3 draft (English Magpie-trained) collapses on Japanese — acceptance 3–4%, slower than baseline (0.69×). The google MTP assistant keeps 35–44% JA acceptance → real +12–23% JA uplift. EN uplift +32–51%.
• ngram never pays for itself on free-form chat (both languages below baseline); don't use it here.
• EAGLE-3 draft is bf16 1.8 GB → at TP=2 it stole enough VRAM that maxlen had to drop 16384→4096 (gmu 0.93). The MTP assistant fits with maxlen 16384 intact at gmu 0.92.
• vLLM 0.21 quantization-inheritance trap does not fire here: with an explicit draft model path, speculative.py passes quantization=None → draft's own config decides (bf16). Inheritance only happens in the model:null MTP-from-target path.

How it was made

1. Bake: QuantizationModifier(targets=Linear, scheme=NVFP4, ignore=[lm_head, re:.*embed.*, re:.*router, re:.*vision_tower.*]) — router, embeddings, lm_head, vision tower kept BF16. Calibration HuggingFaceH4/ultrachat_200k 256×2048 through the multimodal AutoProcessor (bare tokenizer leaves input_global_scale uncalibrated → <pad> spam), pipeline="basic" (gemma4 is fx-untraceable), pure-CPU (~88 min; multi-GPU accelerate dispatch silently corrupts gemma4 activations on no-P2P boxes). MoE experts were permanently unpacked to per-expert Linears via a registered MoECalibrationModule for Gemma4TextExperts with calibrate_all_experts=True.
2. Pad: offline FP4 surgery on the packed checkpoint — per expert, {gate,up}_proj weight+scale padded 704→768 on the output dim, down_proj on the input dim, fill FP4/FP8 0x00 (=+0.0); *_global_scale and all other tensors verbatim. Loss-less (gelu(0)·0=0; padded down-columns multiply zero weights). No re-quant, no GPU. Verified: all 7680+3840 expert tensors 768-aligned, byte size identical to the proven non-QAT pad768.

Notes

• Abliterated (uncensored). Refusal behavior removed upstream — you are responsible for your deployment. Use responsibly and lawfully.
• NVFP4 is Blackwell-specific; it will not run on Ampere/Hopper.
• Gemma is provided under and subject to the Gemma Terms of Use.

Credits

• Base model, QAT-unquantize & abliteration: huihui-ai
• Original model: Google DeepMind (Gemma 4, QAT q4_0)
• NVFP4 quantization, pad768 surgery & judgment: Lna-Lab · Tooling: llm-compressor / vLLM

Support the Base Model Author (huihui-ai)

If you find the abliterated base useful, please support huihui-ai:

• Ko-fi: https://ko-fi.com/huihuiai
• Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge