primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8

4-bit experts, 8-bit around them: 7.9× closer to the reference than NVIDIA's own 24GB build, and 38% faster at concurrency 32.

Mixed NVFP4 / FP8 / Q8_0 build of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 at 46.70 GB, on one 96 GB GPU.
Routed and shared experts store NVFP4 weights and compute in 16-bit. The Mamba projections are FP8 with FP8 activations. The MoE latent projections, attention and the LM head are 8-bit Q8_0. Embeddings, routers, convolutions, norms and the vision tower stay BF16.

BF16 does not fit on one 96 GB card (150 GB), so the reference row is our 8-bit build: every quantized module in Q8_0, 0.54% weight error, 16-bit activations. Drift is the extra negative log-likelihood per token a build assigns to 400 reasoning traces that reference wrote; top-1 agreement is how often its most likely next token is the reference's. The pass is deterministic. BF16 itself could not be scored on one card, so how far the reference sits from BF16 is not measured.

---

Why this quant

• 🎯 7.9× less drift than NVIDIA's 24GB build. +9.6×10⁻³ nats per token, against +75.7 for NVIDIA's 2-bit release and +13.7 for the community NVFP4 build. Top-1 agreement 96.5%, against 92.7% and 96.1%.
• ⚡ 607 tok/s at concurrency 32, against 440 for the 24GB build. The community NVFP4 runs 613 with an FP8 KV cache. Single-stream the 24GB build is faster: 125.5 tok/s against 112.7 here, because it reads fewer bytes per token.
• 📏 Accuracy level with the reference. Knowledge 90.6 and tool calling 82.0, against 90.5 and 82.0 for the reference and 89.7 and 81.0 for the 24GB build, which also finishes fewer answers inside the budget (97.1% against 98.5%).
• 🗜️ 46.70 GB, against 24.08 GB for the 24GB build, 47.78 GB for the community NVFP4 and 150 GB in BF16. On one 96 GB card the command below leaves a 4,074,466-token KV pool at the full 262,144-token window.
• 🧩 The image and kernels NVIDIA's 24GB build needs: the vLLM nightly it names, plus b12x from PyPI.

---

Serve it

hf download primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 --local-dir ./Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models --entrypoint /bin/bash \
  vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989 -c '
  pip install b12x &&
  VLLM_USE_V2_MODEL_RUNNER=1 vllm serve /models/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 \
    --trust-remote-code --max-model-len 262144 --gpu-memory-utilization 0.92 \
    --moe-backend b12x \
    --attention-backend FLASHINFER --mamba-backend flashinfer --mamba-cache-mode align \
    --limit-mm-per-prompt "{\"image\":1,\"video\":0}" \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder \
    --reasoning-parser-plugin /models/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8/ultra_v3_reasoning_parser.py --reasoning-parser ultra_v3'

Checked with exactly this command on an idle card. Boot takes 270 s including the pip install (the weights load in 34 s, 43.96 GiB) and the KV pool is 4,074,466 tokens. What is 17*23? comes back as 391 with the reasoning split off by the parser, a weather question comes back as the parsed call get_weather {"city": "Lisbon"}, and an image of the number 427 is read as 427.

With NVIDIA's LiLiCorr drafter (the --speculative-config from NVIDIA's recipe, 7 tokens), single-stream decode on 24 real prompts goes from 129 to 302 tok/s on this build and from 163 to 377 on the 24GB build (median per request).

---

Measured

One RTX PRO 6000 Blackwell, 96 GB, one card, no tensor parallelism. The 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency 32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache free, two seeds per cell, every build benched on the same card one after another.

Drift and agreement come from 400 traces of the knowledge suite (292,873 completion tokens), scored by each build in one pass; the standard error over traces is 0.4 for this build. Tool calling is three runs per build, knowledge two. KV cache is BF16 for every row except †: its config turns on FP8 KV and vLLM follows it. The 24GB build ran without its speculative drafter; with NVIDIA's FP8 KV setting it measures 127.9 and 442 tok/s.

What this table does not cover:

• Images and video. The suites are text only; the image check above is one prompt.
• Context past 32K. The model's window is 262,144 tokens; we did not test the 4-bit and 8-bit layers there.
• Other GPUs. Only the RTX PRO 6000 Blackwell (sm_120) was tested.

---

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above, the same 1,370 items, auto-scored, no LLM judge.

overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. Read it with finished: overall scores an answer that overran the token budget as wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.

---

What's quantized to what

Nemotron-3.5-Bolt-74B-A9B is 74.9 B parameters including the vision tower, 9 B active per token; 88% of them are routed experts.

ModelOpt MIXED_PRECISION (quantized_layers in config.json and hf_quant_config.json). Q8_0 uses the packing: ggml block layout that NVIDIA's 24GB build uses for the same modules.

---

Licence

OpenMDW License Agreement 1.1, the base model's licence; the full text is in this repository, with NVIDIA's bias, explainability, privacy and safety notes from the original release. This is a modified version of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16: the weights listed above were quantized and config.json carries a quantization_config. Everything else is unchanged. The model and its copyright belong to NVIDIA.

---

_{primitive · more models · inference economics for production LLM systems}