gemma-4-A4B-98e-v7-coder — NVFP4A16 (vLLM)

NVFP4A16 quantization of ManniX-ITA/gemma-4-A4B-98e-v7-coder-it, the loop-fixed code prune of Gemma 4 26B-A4B (128→98 experts/layer, ~20.8B). 2 shards, ~13.4 GB — fits a single 16 GB GPU with room for KV-cache.

NVFP4A16 = 4-bit NVFP4 weights with FP8 (e4m3) block scales and bf16 activations (A16). Quantized with NVIDIA TensorRT-Model-Optimizer (modelopt) main (0.45.0.dev), whose _QuantFusedExperts plugin handles Gemma 4's fused MoE experts. Every exported weight/scale tensor is finiteness-verified before upload.

Serve with vLLM

python -m vllm.entrypoints.openai.api_server \
    --model ManniX-ITA/gemma-4-A4B-98e-v7-coder-NVFP4A16 \
    --served-model-name v7-coder \
    --port 8000 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 65536 \
    --max-num-batched-tokens 8192 \
    --dtype bfloat16 \
    --trust-remote-code \
    --reasoning-parser gemma4 \
    --default-chat-template-kwargs '{"enable_thinking": true}'

Gemma 4 gotchas (all required): --max-num-batched-tokens 8192 (the MM-encoder budget; the default 2048 < max_tokens_per_mm_item and crashes at boot), --max-model-len 65536 (templates may request up to ~49k gen tokens), and the gemma4 reasoning parser for the thinking format. NVFP4 needs a Blackwell/Hopper or Ada GPU on a recent vLLM.

Benchmarks

NVFP4A16 is a deployment format and is not separately benchmarked (cohort policy). The table is the cohort's same-host Q6_K · llama.cpp · greedy reference (temperature 0.0, top_p 1.0, top_k 0), read from summary.json; it is representative of this model's quality. Row-max in bold. This repo = v7-coder.

_{Metrics: GPQA & GSM8K = exact_match flexible-extract · MATH500 = math_verify · ARC & AIME = exact_match · IFEval = prompt_level_strict_acc · HumanEval/+ = pass@1 chat-extract · LCB-55/100 & MultiPL-E = pass@1. 128e uses the lcb_medium_55/100 templates; the prunes use lcb_medium_*_v4 (corrected harness, equivalent task). The all-hard LCB-77 cross-model slice: 128e 79.22% / v7-coder 84.42% / v7-coderx 85.71%.}

Recipe (summary)

98e prune from 128e via the fkbroad code recipe (generate_drop_map_v5: generic_code 3×, targeted_lcb_medium_55 2×, all other targeting 0; target=98, protect_top=16, alpha=2.0, strategy=max, no per-layer floor clamp), then the agentic loop-protection force-keep (agentic_eog, 46 experts, 0/46 dropped — the loop fix that replaces fs2440), then the mandatory shared-FFN α=1.2 upweight, then NVFP4A16 quantization. No targeted_gpqa term and no DERN fold. Full recipe and the calibration-class table are on the bf16 card.

Intended use & limitations

A compact, vLLM-deployable Gemma 4 checkpoint for agentic coding and code reasoning — the balanced code member of the v7-coder cohort (leads LCB-medium + HumanEval; the code-maximal sibling v7-coderx leads the all-hard LCB-77 and HE+). A research prune, not an official Google release; generic_multilingual is de-weighted (0×) and graduate science (GPQA 51.52%) is a budget axis. For llama.cpp/CPU deployment use the GGUF repo.

Lineage

128e → (v4 → v5 → v6-coder code line) → v7 competence-map rebuild → fkbroad code3/lcb2 selection + agentic loop-protection force-keep = v7-coder (loop-fixed; supersedes fs2440) → NVFP4A16. Built and evaluated on the omnimergekit toolchain.