NVIDIA-Nemotron-3.5-Bolt-E4B-NVFP4-GGUF

An NVFP4 GGUF of NVIDIA Nemotron 3.5 Bolt E4B for llama.cpp and LM Studio, built from NVIDIA's official BF16 weights. NVIDIA ships the main model as BF16 and as a Q4_K_M GGUF, and the only official NVFP4 file is the DFlash drafter. So I built the NVFP4 they didn't ship, quality first.

All credit for the model goes to NVIDIA (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16), post-trained from Google's Gemma 4 E4B. Pair it with NVIDIA's own DFlash drafter, nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash.

The short version

Measured on my RTX 5090 with llama.cpp b11856 (official CUDA 13.4 build), greedy, thinking on, median of 9 requests (math, code, chat x 3), all four setups back to back in one sitting.

Files

sha256 in SHA256SUMS. The chat template inside the GGUF is NVIDIA's chat_template.jinja, byte for byte.

Run it

llama.cpp (b11856 or newer), with NVIDIA's DFlash drafter:

llama-server -m NVIDIA-Nemotron-3.5-Bolt-E4B-NVFP4.gguf \
  --spec-draft-model Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf --spec-type draft-dflash \
  --spec-draft-n-max 7 --spec-draft-ngl 999 \
  -ngl 999 -fa on --jinja -c 16384 --parallel 1

Add --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-E4B-BF16.gguf for images and audio (audio input is still marked experimental in llama.cpp).

• Needs llama.cpp b11856 or newer, for NVFP4 per-tensor scales, W4A16 tags and the native Blackwell FP4 kernels. Older builds may load the file but give wrong output.
• Every NVFP4 tensor is tagged W4A16 (general.tensor_extra.prec_a4 = false), so on Blackwell llama.cpp uses 8-bit activations by default. All the numbers here use that setting. GGML_CUDA_MMQ_PREC=q4 forces 4-bit activations: batched matmuls get faster, but KLD rises to 0.090.
• Pass --spec-draft-n-max 7 explicitly (the default is 3). Use --jinja so the Gemma 4 thinking channel parses.
• Context: trained to 65,536 tokens.
• LM Studio: loads and runs on the llama.cpp CUDA 12 runtime 2.64.0 (tested: correct answers). The speed numbers above are from llama-server.

How it was made

1. GPTQ (Frantar et al. 2022) on every decoder linear. Calibration: 621 samples x 2,048 tokens of the BF16 model's own thinking-on generations (math, code, chat, tool use, multilingual, plus 10% plain text). FP32 Hessians, damping 0.01, lazy blocks of 128, and a group-aware column order that keeps every 16-weight NVFP4 block contiguous.
2. Four Over Six (Cook et al., MIT Han Lab + NVIDIA, arXiv 2512.02010). FP4 has no value between 4 and 6, so every 16-weight block tries both scales (map-to-6 and map-to-4) and keeps the one with less error, using the paper's 256 per-tensor scale rule.
   • I make that choice inside the GPTQ loop: when GPTQ reaches each block, both scales run through it and the lower GPTQ loss wins. The paper calls this future work, and on this model it measured better than deciding once beforehand (KLD 0.066 vs 0.070).
   • 42% of blocks picked map-to-4.
   • The file is still standard NVFP4, so llama.cpp's kernels run it unchanged.
3. Sensitive layers at Q8_0, picked from a per-layer KLD sweep:
   • Blocks 22 and 23 were by far the most sensitive, because every layer from 24 to 41 reuses their K/V. Blocks 16-21 and 24 came next.
   • Those 9 blocks plus every attn_k/attn_v stay at Q8_0 (93 of 258 linears).
   • The per-layer input gate and projection stay at Q8_0, since llama.cpp applies no per-tensor scale there.
   • The embeddings stay at Q6_K: Q8_0 measured no better.
   • The 36 never-used K/V tensors of the KV-shared layers are left out.

Bit-exact check: the GGUF bytes match the quantizer's output, and gguf-py's dequant equals the in-memory GPTQ result exactly.

Quality ladder

KLD vs BF16 on 61,440 held-out tokens (wikitext-2 test plus held-out chat and code, never the calibration data):

Tensor map

License

OpenMDW 1.1 (NVIDIA), with the Gemma 4 terms (Apache 2.0) that come with the base model. Same terms as the source checkpoint.