gemma-4-A4B-98e-v7-coderx — NVFP4A16 (vLLM)

NVFP4A16 quantization of ManniX-ITA/gemma-4-A4B-98e-v7-coderx-it, the code-maximal prune of Gemma 4 26B-A4B (128→98 experts/layer, ~20.8B). 2 shards, ~13.4 GB — fits a single 16 GB GPU with room for KV-cache.

NVFP4A16 = 4-bit NVFP4 weights with FP8 (e4m3) block scales and bf16 activations (A16). Quantized with NVIDIA TensorRT-Model-Optimizer (modelopt) main (0.45.0.dev), whose _QuantFusedExperts plugin handles Gemma 4's fused MoE experts. Every exported weight/scale tensor is finiteness-verified before upload.

Serve with vLLM

python -m vllm.entrypoints.openai.api_server \
    --model ManniX-ITA/gemma-4-A4B-98e-v7-coderx-NVFP4A16 \
    --served-model-name v7-coderx \
    --port 8000 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 65536 \
    --max-num-batched-tokens 8192 \
    --dtype bfloat16 \
    --trust-remote-code \
    --reasoning-parser gemma4 \
    --default-chat-template-kwargs '{"enable_thinking": true}'

Gemma 4 gotchas (all required): --max-num-batched-tokens 8192 (the MM-encoder budget; the default 2048 < max_tokens_per_mm_item and crashes at boot), --max-model-len 65536 (templates may request up to ~49k gen tokens), and the gemma4 reasoning parser for the thinking format. NVFP4 needs a Blackwell/Hopper or Ada GPU on a recent vLLM.

Benchmarks

NVFP4A16 is a deployment format and is not separately benchmarked (cohort policy). The table is the cohort's same-host Q6_K · llama.cpp · greedy reference (temperature 0.0, top_p 1.0, top_k 0), read from summary.json; it is representative of this model's quality. Row-max in bold. This repo = v7-coderx.

_{Metrics: GPQA & GSM8K = exact_match flexible-extract · MATH500 = math_verify · ARC & AIME = exact_match · IFEval = prompt_level_strict_acc · HumanEval/+ = pass@1 chat-extract · LCB-55/100 & MultiPL-E = pass@1. 128e uses the lcb_medium_55/100 templates; the prunes use lcb_medium_*_v4 (corrected harness, equivalent task). The all-hard LCB-77 cross-model comparison is the discriminating code slice (v7-coderx 85.71%, cohort-best).}

Recipe (summary)

98e prune from 128e via the code4/lcb3 recipe (generate_drop_map_v5fk: generic_code 4×, targeted_lcb_medium_55 3×, no per-layer floor clamp), then the agentic loop-protection force-keep (46 experts) and the mandatory shared-FFN α=1.2 upweight, then NVFP4A16 quantization. This is the loop-fixed build — it replaces the earlier looping fs2440 prune. v7-coderx is the code-maximal sibling of v7-coder. Full recipe and calibration-class table are on the bf16 card.

Intended use & limitations

A compact, vLLM-deployable Gemma 4 checkpoint for maximal coding throughput and instruction-following — the code-extreme (x) member of the v7-coder cohort. For the broader LCB-medium lead and HumanEval, use v7-coder, which leads those slices (GPQA ≈ 51 on both siblings — neither recovers science). A research prune, not an official Google release; generic_multilingual is de-weighted (0×) and graduate science (GPQA) is a budget axis — at 51.01% it is well below the unpruned 128e (67.17%). For llama.cpp/CPU deployment use the GGUF repo.

Lineage

128e → (v4 → v5 → v6-coder code line) → v7 competence-map rebuild → code4/lcb3 selection + agentic loop-protection force-keep = v7-coderx → NVFP4A16. Built and evaluated on the omnimergekit toolchain.