NVIDIA Nemotron 3.5 Bolt 74B-A9B — GGUF, 3- to 8-bit, from the BF16 weights

GGUF quants of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 for llama.cpp: a 74B Mamba-2 / Transformer hybrid with LatentMoE (9B active) and a C-RADIO v4-H vision encoder.

NVIDIA's own GGUF is a single 2-bit file trained for that precision (the 24 GB QAD checkpoint). These files are quantized from the full-precision BF16 weights instead, at 3 to 8 bits, for machines with more memory: 2 files from 28 to 45 GiB, plus the vision projector.

Which file

The routed experts are ~89% of the weights, so every file runs on a GPU of any size by keeping some experts in system RAM. Rows: total memory you need = VRAM + RAM. Pick the largest file whose RAM column fits your machine.

Quick pick (largest file that fits, keeping 8 GB of RAM for everything else):

Being uploaded: Q8_0, Q6_K, Q5_K_M, IQ4_XS, Q3_K_M (each one after its quality check; this card updates as they land).

Download:

pip install -U huggingface_hub   # provides the `hf` command
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF --include "Q4_K_M/*" --local-dir .   # 45 GiB in 2 parts
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF NVIDIA-Nemotron-3.5-Bolt-74B-A9B-IQ3_XXS.gguf --local-dir .   # 28 GiB
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf --local-dir .   # vision (BF16 version: mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16.gguf)

Flags are for a recent llama.cpp; with the default --fit on, llama.cpp chooses the offload itself and you can leave -ngl / -ncmoe out (see "Run"). VRAM figures count weights + 32K context + compute buffers. VRAM numbers are computed from the files (weights + KV + compute estimate); measured values replace them after the B70 runs.

Quality

KL divergence and top-1 token agreement against the Q8_0 file, and perplexity, on wikitext-2 test (32,768 tokens, 2048-token windows). Q8_0 is the reference because the BF16 model (138 GiB) does not fit the test machine; Q8_0 is normally within noise of BF16.

Compared with NVIDIA's 24 GB file

NVIDIA's 24 GB GGUF is 2-bit, recovered with quantization-aware distillation (QAD): IQ2_XXS routed and shared experts, NVFP4 Mamba-2 in/out projections, Q8_0 attention, LatentMoE projections, embedding and output. It fits a 24 GB card whole at 32K context; NVIDIA's own llama.cpp recipes are for 32 GB+ GPUs (64K context, q8_0 KV cache) and 48 GB+ (256K). The files here trade memory for fidelity to the BF16 weights. Same measurement as above (KL and top-1 vs our Q8_0, wikitext-2 test):

NVIDIA's file is quantized from its own QAT-trained checkpoint, so its KL against our BF16-based Q8_0 includes that retraining, not only rounding error; PPL compares both on equal terms.

Memory: weights + 32K context + compute buffers (computed; on the B70 NVIDIA's file measured 22.9 GiB with its LiLiCorr drafter at 32K). Speed on the B70: NVIDIA's file 43 tok/s decode (Mesa 26.2); ours in the speed table below.

Speed: one Intel Arc Pro B70 (32 GB)

llama.cpp Vulkan, Mesa 26.2.5, Ryzen 9 5950X, 64 GB system RAM. Flags from the 8 / 12 / 24 GB rows above (the B70 then uses only that much VRAM). Prompt 512 tokens; generation 128 tokens at empty and 16K context.

Run

Needs llama.cpp b11669 or newer (October 15, 2026). Bolt's MoE layers have different expert counts (240 to 496), and llama.cpp loads that only since #30520, which also added the vision encoder. Older builds stop at load time with a tensor shape error, for text too.

# any GPU (CUDA, Vulkan, SYCL, Metal) or CPU; --fit (default on) picks the offload for your VRAM
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf \
  -c 32768 -fa on --temp 1.0 --top-p 0.95 --jinja

# or set the offload yourself (example: 24 GB card, flags from the table)
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf -ngl 99 -ncmoe 62 -c 32768 -fa on --temp 1.0 --top-p 0.95 --jinja

# CPU only
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf -ngl 0 -c 32768 --temp 1.0 --top-p 0.95 --jinja

# one image from the command line
llama-mtmd-cli -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf \
  --temp 1.0 --top-p 0.95 --image photo.jpg -p "Describe this image."

Split files (-00001-of-0000N.gguf): point -m at the first part.

Context: NVIDIA's card gives up to 256K tokens (262,144, so -c 262144); its compression recovery trained at 8K, then 128K. Only 8 of the 88 blocks are attention, so the KV cache is small (about 1 GiB at 128K, f16); the 40 Mamba blocks keep a fixed 160 MiB state per sequence.

Sampling: NVIDIA recommends temperature 1.0, top_p 0.95 (also in its generation_config.json), with reasoning on or off. Keep --jinja: the reasoning switch and tool calls come from the model's chat template.

Thinking on/off

Reasoning is on by default (the chat template's enable_thinking, default true). Turn it off for the whole server, or per request:

# server-wide
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf ... --jinja --chat-template-kwargs '{"enable_thinking": false}'

# per request (OpenAI-compatible API; Python client: extra_body={"chat_template_kwargs": {...}})
curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Why does binary search take logarithmic time?"}],
  "chat_template_kwargs": {"enable_thinking": false}
}'

Medium reasoning effort: {"enable_thinking": true, "medium_effort": true} (the template appends {reasoning effort: efficient} to the last user message). Reasoning and the answer share the max_tokens / -n budget; NVIDIA's examples use 16384. By default the template drops the reasoning of earlier turns from the history (truncate_history_thinking, default true).

Tools and vision

• Tool calling: pass tools in the OpenAI-style request to llama-server --jinja. The chat template writes calls as Qwen3-Coder-style XML (<tool_call><function=name><parameter=...>); NVIDIA's vLLM recipe parses them with --tool-call-parser qwen3_coder. NVIDIA's 24 GB GGUF card has a complete tool-calling client for llama-server. We have not tested tool calls with these files yet.
• Images: load the projector with --mmproj (Q8_0, or BF16 for full precision) and send images as image_url content parts, or use llama-mtmd-cli as above. NVIDIA also lists video input (mp4, up to 2 minutes); we have not tried video with llama.cpp.
• Languages (NVIDIA's card): English, French, Spanish, Italian, German, Japanese, Hindi, Korean, Brazilian Portuguese and Chinese. NVIDIA's metadata tags the first six; this card uses the same tags.

Intel Arc (B70, B60, B580)

Use the Vulkan backend with Mesa 26.2 or newer: on the B70, Mesa 25.3 decodes this model about 40% slower (measured with NVIDIA's 24 GB file: 25 vs 43 tok/s). SYCL works too. Optional: NVIDIA's LiLiCorr draft model (...-24GB-LiLiCorr-GGUF) for speculative decoding: --spec-type draft-dflash --spec-draft-model <lilicorr.gguf> --spec-draft-ngl 99 --spec-draft-n-max 6 (+27% on code with the 24 GB file, no gain on prose; acceptance with these BF16-based files not measured yet).

How these were made

1. BF16 safetensors to GGUF with convert_hf_to_gguf.py from llama.cpp master plus a fix for this checkpoint (riceharvest/llama.cpp bolt-convert): Bolt's Puzzle config sits inside the Omni wrapper (llm_config), its MoE blocks have 240-496 experts each, and its hidden_act describes the MLP. Tensor names and metadata match NVIDIA's own GGUF (expert_count 496, moe_latent_size 1024).
2. Importance matrix from a Q6_K file quantized without one (its experts fit this machine's RAM; Q8_0 would page from disk for hours): 152 chunks of 512 tokens of calibration_datav3 (mixed prose, code, math, multilingual) + wikitext-2 train + C++/Python source. The wikitext test split used for evaluation is not in it.
3. llama-quantize with per-tensor overrides. The routed experts take the file's type; the small, always-on tensors are kept higher (NVIDIA's own QAT file also keeps attention and the LatentMoE projections at Q8_0):

Scripts: in this repo's scripts/ folder (convert, imatrix, quantize, eval, per-tier bench, size calculator).

Credits and license

• Model: NVIDIA (NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16). See NVIDIA's model card for intended use, training data, evaluation, bias, safety and privacy notes.
• License: OpenMDW-1.1, same as the original. A copy is in LICENSE.OpenMDW-1.1, and NVIDIA's notices are kept in NOTICE.md.
• Runtime: llama.cpp (ggml-org). Calibration text: bartowski's calibration_datav3.
• Quantized by @imdariotoo (GitHub riceharvest).