Gemma 4 12B-it — FP8 dynamic
Self-quantized FP8 (dynamic) of google/gemma-4-12B-it — Google's encoder-free omni model (text + image + audio + video). Quantized and benchmarked on an NVIDIA DGX Spark (GB10, sm_121a).
TL;DR: 13 GB on disk (from 23 GB BF16), 15.9 tok/s on a GB10 via vLLM, all four modalities intact. Data-free — no calibration needed.
Benchmark (GB10 / DGX Spark, vLLM 0.22.1 native, single-stream decode, warm)
| Format | Disk | tok/s (EN/ZH) | Omni |
|---|---|---|---|
| BF16 | 23 GB | 7.7 | yes |
| FP8 dynamic (this) | 13 GB | 15.9 | yes |
| NVFP4 W4A16 | 7.7 GB | 24.9 | yes |
If you want the smallest + fastest build, see the sibling NVFP4 weight-only repo. FP8 is the conservative choice (dynamic activations, no calibration, widest kernel support).
Accuracy (MMLU + TMMLU+) — near-lossless on both languages
I scored all three formats on MMLU (English, 57 subjects) and TMMLU+ (Traditional Chinese, 66 subjects) with lm-evaluation-harness, 5-shot, chat template applied, limit=30 (N ≈ 1,710 EN / 1,980 TC, ±~1.0 pt), through transformers:
| Format | MMLU (EN) | TMMLU+ (TC) | EN drop | TC drop |
|---|---|---|---|---|
| BF16 | 78.30% | 47.21% | — | — |
| FP8 dynamic (this) | 77.95% | 46.97% | −0.35 | −0.24 |
| NVFP4 W4A16 | 75.56% | 41.24% | −2.74 | −5.97 |
FP8 is the accuracy-preserving choice. Near-lossless on both languages (within ~0.4 pt) and symmetric — while weight-only NVFP4 drops Traditional Chinese by ~6 points. If you can spare the extra disk/bandwidth over NVFP4 and care about non-English quality, FP8 is the safer pick. limit=30, single model — indicative. Full writeup.
Quantization recipe
llmcompressor, scheme FP8_DYNAMIC, data-free (no calibration data). Ignore list keeps the head and the multimodal projectors in BF16:
QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC",
ignore=["lm_head", "re:.*embed_vision.*", "re:.*embed_audio.*"])
Serving (vLLM)
Needs vLLM with native Gemma4UnifiedForConditionalGeneration (~0.22.x / main) and the TRITON_ATTN backend (Gemma 4 has heterogeneous head dims: head_dim 256 x 16 = 4096 != hidden 3840):
VLLM_ATTENTION_BACKEND=TRITON_ATTN \
vllm serve coolthor/gemma-4-12B-it-FP8-dynamic --max-model-len 4096
Environment (exact versions — this model is version-sensitive)
This is a brand-new arch on a brand-new GPU, so the toolchain matters more than usual. The versions I actually ran:
| Component | Version | Why it matters |
|---|---|---|
| vLLM | 0.22.1rc1.dev124 (main, post-PR) |
Needs the native Gemma4UnifiedForConditionalGeneration class, which only landed around 0.22.x/main. On an older vLLM it falls back to the generic transformers backend, which mishandles Gemma 4's non-square attention and crashes on o_proj. |
| transformers | 5.10.1 |
First release that knows model_type: gemma4_unified. Older transformers can't even load the config. |
| torch | 2.11.0+cu130 |
The one that bit me. vLLM main pins torch==2.10, but its _C.abi3.so was compiled against 2.11+cu130 — installing the pinned 2.10 (and pip silently pulling the CPU wheel on arm64) gives an undefined symbol import error and a CPU-only build. Force-align: pip install --force-reinstall --no-deps torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130. |
| compressed-tensors | bundled with llmcompressor | Reads the FP8 weight format. |
| GPU / arch | DGX Spark GB10, sm_121a, CUDA 13.x |
The torch-ABI dance above is specific to building vLLM from source for sm_121. |
| Attention backend | VLLM_ATTENTION_BACKEND=TRITON_ATTN |
Required, not optional — see the head-dim note above. |
On a normal CUDA GPU (Hopper/Ada/Blackwell desktop) you don't need the torch-ABI overlay — that pain is specific to building vLLM from source for sm_121. A recent pip install vllm (with the native Gemma4Unified class) plus VLLM_ATTENTION_BACKEND=TRITON_ATTN is enough. FP8 is data-free, so the quantization recipe above is the whole reproduce step — no calibration set needed.
Validation (GB10, transformers)
- Text: coherent EN + ZH.
- Image: accurately described a studio-podcast photo (cat + Shiba Inu, headphones, studio mics, latte-art mug).
- Audio: understood a LibriSpeech clip (Mr. Quilter / apostle / middle classes).
- Video: correctly described a night-street clip.
Credits
- Base model:
google/gemma-4-12B-it(Apache 2.0, Google DeepMind) - Quantization:
llmcompressor+compressed-tensors - Quantized & benchmarked by coolthor on a DGX Spark (GB10)
Support
One-person effort on a single DGX Spark, no sponsor. If it saved you time, a coffee ☕ is appreciated.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
Provenance
This section documents release efdc1c876c34a8bd0cfd90886818fba7277963d9, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
Source lineage
| Source repository | Revision used for this build | Role |
|---|---|---|
| google/gemma-4-12B-it | not recorded | BF16 input checkpoint |
Build details
Tools and versions: llm-compressor and compressed-tensors, exact build-time versions/commits not recorded. Published config records transformers 5.10.1; card records torch 2.11.0+cu130. Serving vLLM versions are not quantizer versions.
The model was quantized with FP8_DYNAMIC without calibration data, with lm_head, re:.embed_vision. and re:.embed_audio. left unquantized.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Build script (copied from the existing card, not re-run): Because it references the repo by name without a pinned commit, the exact upstream commit used at build time is unknown.
QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC",
ignore=["lm_head", "re:.*embed_vision.*", "re:.*embed_audio.*"])
The serialized quantization settings are in recipe.yaml.
Weight file SHA-256 checksums
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
model.safetensors |
13024127312 | 0dde463f314286b4c3ed6fa0f6bda8f91b26fcb97ac35e7a829e4350db7e83ed |
Verifying the files
Download the desired files from revision efdc1c876c34a8bd0cfd90886818fba7277963d9 and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Base revision; Exact quantizer versions/commits and full invocation.
Sources: existing model card and HF metadata.