Huihui-gemma-4-26B-A4B-it-abliterated FP8-Dynamic
vLLM-compatible FP8-Dynamic quantization of huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated.
The original BF16 release (~50 GB) is not directly servable by vLLM at production speed — --quantization fp8 runtime path is ~6× slower than pre-quantized FP8. This repo fills that gap: 27 GB FP8 weights, drop-in for vLLM, multimodal vision retained, abliteration profile preserved.
Smoke test (DGX Spark / GB10)
| Modality | Result |
|---|---|
| Text English (haiku) | ✅ coherent |
| Text 繁體中文 | ✅ humor + structure preserved |
| Vision (古風美女 portrait) | ✅ correctly described hanfu, bamboo, pose, expression |
| Audio | N/A — Gemma 4 26B-A4B-it has audio_config: null (vanilla design, not quant artifact) |
Speculative-decoding bench (DGX Spark, GB10, batch=1, T=0.0)
Tested pairing this model as MTP target with the vanilla google/gemma-4-26B-A4B-it-assistant draft model:
| Config | Acceptance | Throughput |
|---|---|---|
| Baseline (no spec) | n/a | 39.3 tok/s |
| MTP, num_speculative_tokens=4 | 40% token-level (per-pos: 65 / 43 / 29 / 21) | 52.4 tok/s (+33%) |
| MTP, num_speculative_tokens=1 | 69% | 52.6 tok/s (+34%) |
Verdict: MTP gives a real +33% gain even with vanilla draft head. num_speculative_tokens=1 is recommended — cleaner per-step acceptance (69% vs 40% token-level for n=4), same throughput. The per-position decay (65% → 21% over 4 positions) is consistent with huihui's abliteration shifting the body's prediction distribution; n=1 only uses the high-confidence first position.
For comparison, vanilla Gemma 4 + MTP achieves 108 tok/s on the same hardware (Part 27 reference). The ~50% gap is the "abliteration tax" — likely from a combination of FP8 calibration on shifted weights and MoE routing imbalance after abliteration. If you need raw speed without abliteration, use the vanilla google/gemma-4-26B-A4B-it directly.
vLLM serving
Recommended (with MTP, +34%)
vllm serve coolthor/Huihui-gemma-4-26B-A4B-it-abliterated-FP8-Dynamic \
--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":1}' \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.65 \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image":0,"audio":0,"video":0}' \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--trust-remote-code
Note: MTP requires vLLM with PR #41745's gemma4_mtp.py integration. Tested with the vllm/vllm-openai:gemma4-0505-arm64-cu130 image.
Usage (client side)
⚠️ This is an instruction-tuned model. Always use the chat completions endpoint.
curl -s -X POST http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "huihui-gemma4",
"messages": [{"role": "user", "content": "Write a haiku about an old API."}],
"max_tokens": 80,
"temperature": 0.7
}'
Do NOT use /v1/completions with a raw prompt — it bypasses the Gemma chat template (<start_of_turn>user ... <end_of_turn>\n<start_of_turn>model\n) and the model will fall off-distribution and loop garbage tokens. This is not abliteration / FP8 / MTP related; vanilla google/gemma-4-26B-A4B-it behaves the same way under /v1/completions. If you must use the completions endpoint, apply the chat template manually.
Quantization recipe
Used llmcompressor with FP8_DYNAMIC scheme. Critical ignore list — note re:.*router.* in particular: omitting this entry quantizes MoE router weights and breaks expert dispatch (verified by trial in our own testing):
ignore = [
"re:.*router.*", # MoE router — must NOT quantize
"lm_head",
"re:.*embed_tokens.*",
"re:.*norm.*", "re:.*layernorm.*", "re:.*layer_norm.*",
"re:.*rmsnorm.*", "re:.*rms_norm.*",
"re:.*conv1d.*", "re:.*linear_attn.*",
"re:visual.*", "re:model.visual.*",
"re:.*patch_embed.*", "re:.*vision.*", "re:.*image.*",
"re:.*video.*", "re:.*projector.*", "re:.*merger.*",
"re:.*mlp.gate$", "re:.*shared_expert_gate.*",
"re:.*embed_audio.*", "re:.*embed_vision.*",
"re:.*audio_tower.*", "re:.*audio_projector.*",
]
Quantization run: 2.9 minutes on GB10 (data-free pipeline, 30 MoE module calibration, FP8 scale derivation).
Toolchain notes
llmcompressor stable releases pin transformers <= 4.57.6, but Gemma4ForConditionalGeneration requires transformers 5.0+. Workaround: patch llmcompressor/entrypoints/utils.py to remove deprecated use_auth_token=... kwargs (8 occurrences) before reinstalling editable.
Credits
- Base abliteration: huihui-ai
- Original Gemma 4:
google/gemma-4-26B-A4B-it - MTP draft model:
google/gemma-4-26B-A4B-it-assistant - Quantization toolchain:
vllm-project/llm-compressor
License
Apache 2.0, inherited from base model and Gemma usage terms.
☕ If this saved you GPU hours, you can buy me a coffee.
📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
Provenance
This section documents release ef428e5d1b9bc0b101546ec517c6fa205bb5ff27, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
Source lineage
| Source repository | Revision used for this build | Role |
|---|---|---|
| huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated | not recorded | Immediate BF16 input; upstream already abliterated |
Build details
Tools and versions: llm-compressor, exact build-time version/commit not recorded. Existing card documents removing deprecated authentication kwargs from entrypoints/utils.py for transformers 5 compatibility.
The model was quantized with FP8_DYNAMIC, with MoE routers, the LM head, embeddings, norms, multimodal modules and routing gates left unquantized, as specified by the repository recipe.yaml. The recovered local script path is /tmp/quant_huihui_gemma4.py, but the script body was not recovered.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Build script (copied from the existing card, not re-run): Because it references the repo by name without a pinned commit, the exact upstream commit used at build time is unknown.
ignore = [
"re:.*router.*", # MoE router — must NOT quantize
"lm_head",
"re:.*embed_tokens.*",
"re:.*norm.*", "re:.*layernorm.*", "re:.*layer_norm.*",
"re:.*rmsnorm.*", "re:.*rms_norm.*",
"re:.*conv1d.*", "re:.*linear_attn.*",
"re:visual.*", "re:model.visual.*",
"re:.*patch_embed.*", "re:.*vision.*", "re:.*image.*",
"re:.*video.*", "re:.*projector.*", "re:.*merger.*",
"re:.*mlp.gate$", "re:.*shared_expert_gate.*",
"re:.*embed_audio.*", "re:.*embed_vision.*",
"re:.*audio_tower.*", "re:.*audio_projector.*",
]
The serialized quantization settings are in recipe.yaml.
Weight file SHA-256 checksums
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
model.safetensors |
28641731036 | 230b16590be32cce1aea597e5d5636a5c8e62b20c3d6cd710798ad9cb6ec8ad4 |
Verifying the files
Download the desired files from revision ef428e5d1b9bc0b101546ec517c6fa205bb5ff27 and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Base revision; Exact compressor version/commit and local patch; Complete build script/command.
Sources: existing model card and HF metadata.