gemma-4-31b-he1-it-NVFP4A16
NVFP4-A16 quantization (4-bit FP4 weights, BF16 activations) of ManniX-ITA/gemma-4-31b-he1-it — a partial L0–L3 Q-head prune of Gemma 4 31B-it with lstsq O-projection heal.
For the full pruning recipe (per-tensor diff, v1 vs v2 SHA evidence, BOS gotcha, refit methodology, llama.cpp Q4_K_M legacy eval), see the BF16 base card. This card focuses on the NVFP4A16 quant for vLLM serving.
Quant recipe
- Quantizer:
omnimergekit/scripts/quantize_any.py --method nvfp4a16 - Engine:
nvidia-modelopt==0.43.0withNVFP4_DEFAULT_CFG - Calibration: 128 samples from
tatsu-lab/alpaca, max_length=512 - Excluded modules:
*vision_tower*,*embed_vision*,*embed_audio*(kept BF16 for vLLM'sgemma4_mmloader) - Producer hardware: vast.ai RTX 6000 Ada 48 GB pod
- Producer stack: transformers 5.5.0, torch 2.10.0+cu128, safetensors 0.7.0, modelopt 0.43.0
- Date: 2026-05-17
The pruned L0–L3 q_proj/o_proj projections quantize the same way as the unpruned L4–L59 — NVFP4-A16 is applied uniformly across the model. The partial-prune footprint is preserved in the quantized weights.
Eval — canonical 9-bench cohort (NVFP4A16, vLLM 0.20.2 stock, greedy)
Greedy sampler (temperature=0, top_p=1, top_k=0, do_sample=False), thinking_token_budget=12288, max_gen_toks=16384. Bench definitions: omnimergekit/eval/templates/. 128e / v4 columns reuse the published canonical baseline from the v5-it card.
| Bench (n) | 128e ref | 98e v4 | 98e v5-coder | 31B-it (base) | 31B-he1 (this model) |
|---|---|---|---|---|---|
| GPQA Diamond (198) | 73.23 % | 69.19 % | 68.69 % | 81.31 % | 84.34 % |
| GSM8K-100 | 91.00 % | 86.00 % | 86.00 % | 93.00 % | 92.00 % |
| MATH-500-100 | 89.00 % | 89.00 % | 92.00 % | 97.00 % | 95.00 % |
| AIME 2024 (30) | 36.67 % | 36.67 % | 36.67 % | 76.67 % | 76.67 % |
| IFEval-100 (prompt_strict) | 95.00 % | 93.00 % | 94.00 % | 96.00 % | 96.00 % |
| HumanEval-164 chat | 96.95 % | 96.95 % | 98.17 % | 97.56 % | 98.17 % |
| HumanEval+ chat (164) | 92.07 % | 91.46 % | 92.68 % | 92.07 % | 93.90 % |
| LCB-medium-55 (v4 split) | 87.27 % | 78.18 % | 85.45 % | 96.36 % | 96.36 % |
| ARC-Challenge chat (1172) | 95.99 % | 95.99 % | 95.31 % | 98.04 % | 97.61 % |
Bold = best in row. Source: L40 pod 37006213, vLLM 0.20.2 stock + --reasoning-parser gemma4 + thinking_token_budget=12288 + Fix-A reasoning_content fallback, greedy sampler, eval window 2026-05-18 / -19. 128e / v4 / v5-coder columns reuse their respective published baselines on the same pinned stack.
What the he1 column says
The null-hypothesis framing was falsified positively: the L0–L3-only modification doesn't just match base — it lifts the two hardest benches (GPQA Diamond, HumanEval+) and ties two more (HumanEval, LCB-medium-55, AIME, IFEval).
- GPQA Diamond +3.03 pp (84.34 % vs 81.31 %) — the L0–L3 partial prune slightly favours multi-domain reasoning paths rather than degrading them. This is the largest delta on the table.
- HumanEval+ +1.83 pp vs base (93.90 % vs 92.07 %); +1.22 pp over v5-coder's 92.68 % — he1 is the new best in the cohort on the harder HE+ split.
- LCB-medium-55 96.36 % — tied with base for best in cohort. Both the unmodified base and the partial-prune land identical 53/55 scores; the prune doesn't perturb the v4 LCB-medium answer paths.
- HumanEval 98.17 % (+0.61 pp vs base; tied with v5-coder), AIME 2024 76.67 % (tied with base), IFEval 96.00 % (tied) — three apples-to-apples ties or near-ties on instruction-following and code-generation.
- GSM8K −1.00 pp, MATH-500 −2.00 pp, ARC −0.43 pp — all inside their per-bench stderr (±1.7 – ±2.7 pp); not statistically distinguishable from base.
Practical reading: the partial L0–L3 prune (~0.83 % attention savings) doesn't regress on any canonical bench, lifts GPQA Diamond and HumanEval+, and ties LCB-medium-55 with base. It is the strongest Gemma 4 variant in this cohort on the two hardest benches (GPQA Diamond, HumanEval+).
Usage with vLLM
python -m vllm.entrypoints.openai.api_server \
--model ManniX-ITA/gemma-4-31b-he1-it-NVFP4A16 \
--port 8099 --tensor-parallel-size 1 \
--gpu-memory-utilization 0.70 \
--max-model-len 32768 \
--max-num-batched-tokens 8192 \
--max-num-seqs 4 \
--dtype bfloat16 \
--trust-remote-code \
--reasoning-parser gemma4 \
--default-chat-template-kwargs '{"enable_thinking": true}'
Per-request:
{"thinking_token_budget": 12288, "temperature": 0.0, "top_p": 1.0, "top_k": 0}
VRAM
| GPU | Runs? | Memory (peak) | Notes |
|---|---|---|---|
| RTX 3090 24 GB | ❌ | OOM | Single-GPU at 32 k ctx exceeds 24 GB — use TP=2 |
| RTX 4090 24 GB | ❌ | OOM | Same — TP=2 required |
| RTX 6000 Ada 48 GB | ✅ | ~32 GB | Single-GPU, 32 k ctx |
| L40 / L40S 48 GB | ✅ | ~32 GB | Single-GPU, 32 k ctx |
| A100 / H100 80 GB | ✅ | ~32 GB | Single-GPU, 65 k ctx headroom |
| 2× RTX 3090 / 4090 | ✅ | ~16 GB / GPU | TP=2, 32 k+ ctx |
The he1 partial prune saves ~0.83 % of attention parameters (4 layers × 12.5 % Q-prune / 60 layers); VRAM footprint is essentially identical to the 31B-it base NVFP4A16.
Related models
| Model | Description |
|---|---|
| gemma-4-31b-he1-it | BF16 base — full design notes, per-tensor diff, llama.cpp Q4_K_M legacy eval |
| gemma-4-31b-he1-it-GGUF | GGUF tier sweep (llama.cpp / ollama) |
| Gemma-4-31B-it-NVFP4A16 | Unpruned 31B baseline NVFP4A16 — the apples-to-apples comparison target |
| google/gemma-4-31B-it | Upstream BF16 base — the unpruned reference |
Source
- BF16 base:
ManniX-ITA/gemma-4-31b-he1-it(partial L0–L3 prune ofgoogle/gemma-4-31B-it) - Pruning recipe:
omnimergekit/recipes/gemma4_31b/prune_local_heal.py - Quant script:
omnimergekit/scripts/quantize_any.py - Eval orchestrator:
omnimergekit/scripts/eval_suite_vllm.sh - Eval protocol:
omnimergekit/eval/EVAL_PROTOCOL.md
License
Inherits the Gemma license from the upstream Gemma 4 base.
Acknowledgements
- Google for the base Gemma 4 31B-it model
- OmniMergeKit for the prune-and-heal recipe and eval toolkit
- The vLLM and NVIDIA ModelOpt teams for the NVFP4A16 serving / quantization pipeline