gemma-4-31B-it-speculator.eagle3-NVFP4 (NVFP4)

NVFP4 quantization of RedHatAI/gemma-4-31B-it-speculator.eagle3 — RedHat / Neural Magic's official EAGLE-3 speculator drafter for Gemma 4 31B (dense).

What this is

Drop-in replacement for the BF16 drafter, 3× smaller (4.5 GB → ~1.5 GB) and ~1.5× faster per draft step on Blackwell with native FP4 tensor cores. Targets the same verifier model as the BF16 source.

Use it with vLLM

vllm serve RedHatAI/gemma-4-31B-it-NVFP4 \
  --tensor-parallel-size 1 \
  --speculative-config '{
    "model": "AEON-7/gemma-4-31B-it-speculator.eagle3-NVFP4",
    "num_speculative_tokens": 3,
    "method": "eagle3"
  }' \
  --max-num-seqs 8 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --enable-prefix-caching

Verifier can be any Gemma 4 31B (dense) variant — base, instruct, NVFP4, or fine-tunes (abliterated, domain-tuned, etc.). EAGLE drafters are architecture-bound, not weights-bound; output distribution is provably the verifier's.

Quantization recipe

Performance expectations

Acceptance lengths (relative to BF16 source — RedHat's published numbers):

Roughly 8-12% acceptance loss vs BF16, more than offset by per-step speedup on Blackwell native FP4 hardware.

Spark / DGX Spark deployment

Tested on NVIDIA DGX Spark (GB10, sm 12.1, 128 GB unified memory) using the ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4:latest image (eugr nightly with sm_120-compiled FlashInfer CUTLASS + VLLM_CUTLASS NVFP4 kernels).

Single-stream wall-clock: 2.0-2.5× speedup over no spec decode on chat workloads.

Files

• model.safetensors — NVFP4 quantized drafter weights (~1.5 GB)
• config.json — Eagle3 speculator config (carries verifier reference)
• config.py — Custom Eagle3SpeculatorConfig class (custom_code, required for trust_remote_code=True)
• tokenizer.json, tokenizer_config.json — Verifier tokenizer (Gemma 4)
• hf_quant_config.json — ModelOpt NVFP4 quantization metadata
• modelopt_state.pt — Full modelopt state for re-export

License

Apache 2.0 (matches base model). NVFP4 quantization is a derivative work contributed under the same terms.

Provenance

Created by quantizing RedHatAI/gemma-4-31B-it-speculator.eagle3 with NVFP4_DEFAULT_CFG calibrated against RedHatAI/gemma-4-31B-it-NVFP4 outputs. Methodology adapted from RedHat AI's published Gemma 4 NVFP4 target recipe + standard EAGLE-3 calibration practice.

See also

• BF16 source: RedHatAI/gemma-4-31B-it-speculator.eagle3
• Verifier (NVFP4): RedHatAI/gemma-4-31B-it-NVFP4
• EAGLE-3 paper: arXiv:2503.01840
• Speculators library: vllm-project/speculators

---

☕ Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:

Become a member on Patreon → patreon.com/cw/AeonForge7/membership

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.