Nemotron 3.5 Bolt 74B-A9B in EXL3, 2.0 bits per weight
These are EXL3 weights of NVIDIA's Nemotron 3.5 Bolt 74B-A9B at 2.0 bits per weight: 2.19 bits on average over the text weights, 21.7 GB. We converted them from NVIDIA's BF16 checkpoint, nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 at revision bca0a8cb, with exllamav3 1.6.2 and a patch for Bolt's layer layout (in convert/). The routed experts are at 2.0 bits. The Mamba2, attention, latent and shared-expert projections are at 4 bits, and the output head at 6. The embedding stays in BF16. The RADIO image tower and its projector are in the files unchanged, in BF16 (1.7 GB); TandemLLM serves text only for now and does not load them.
We serve these weights with TandemLLM, our inference engine for one DGX Spark. Bolt is 88 layers: 40 Mamba2, 40 mixture-of-experts with 240 to 496 experts each (9 a token, plus one shared expert) and 8 attention layers, 262,144 tokens of context.
Quality of every level
We compared each pack with BF16 on 8 rows of wikitext-2, 1,024 tokens each (8,192 positions). The KL, top-1 and perplexity columns are from TandemLLM's engine against BF16's logits on the same tokens. BF16's perplexity on these rows is 5.97. The model_diff columns are exllamav3's eval/model_diff.py on 100 rows of 2,048 tokens, where BF16's perplexity is 5.66; that run is in quality/ of each repository. GB counts the text weights only.
| level | bpw | GB | KL | top-1 | ppl | KL (model_diff) | top-1 (model_diff) |
|---|---|---|---|---|---|---|---|
| 2.0 | 2.19 | 21.7 | 0.170 | 84.0 % | 6.81 | 0.0881 | 87.6 % |
| 2.5 | 2.64 | 25.8 | 0.094 | 87.3 % | 6.34 | 0.0492 | 90.6 % |
| 3.0 | 3.09 | 30.0 | 0.061 | 89.7 % | 6.21 | 0.0321 | 92.2 % |
| 3.5 | 3.55 | 34.1 | 0.043 | 91.1 % | 6.12 | 0.0236 | 93.3 % |
| 4.0 | 4.00 | 38.2 | 0.034 | 92.7 % | 6.10 | 0.0197 | 93.8 % |
| 5.0 | 5.00 | 47.4 | 0.016 | 94.8 % | 6.00 | 0.0084 | 96.0 % |
| 6.0 | 6.00 | 56.6 | 0.009 | 96.2 % | 5.98 | 0.0047 | 97.1 % |
| NVIDIA IQ2 GGUF | 2.38 | 22.0 | 0.259 | 79.6 % | 7.48 |
Below 4.0 we keep every projection outside the routed experts at 4 bits. On the 2.0 pack that change took KL from 0.379 to 0.170. From 4.0 up every projection is at the level.
Speed on a DGX Spark
TandemLLM 0.10.0 on one DGX Spark (GB10, 128 GB), the box running nothing else. One request at a time, 512 greedy tokens, thinking off, a 32,768-token window, a fresh server for every run. Every level ran three plain runs. 2.0, 4.0 and NVIDIA's file also ran three runs with NVIDIA's LiLiCorr drafter plus TandemLLM's lookup (StairCut picks the block size), alternating with the plain ones; for the other levels the drafted speed is not measured, see 2.0 and 4.0. Each cell is the mean and ± half the range of the runs. Plain is the mean over the five asks. Edit and copy are asks whose answer repeats much of the prompt. On every ask, the text with LiLiCorr and the lookup was the same as the plain text. On 4.0 and NVIDIA's file, TandemLLM 0.10.0 gets no speed from LiLiCorr and the lookup: its step-size rule (StairCut) keeps plain steps on every ask, so those cells are plain decoding. We have not found the cause yet; TandemLLM 0.10.0's profile for NVIDIA's file drafts with the lookup alone.
| tok/s | plain | LiLiCorr, prose | chat | code | edit | copy |
|---|---|---|---|---|---|---|
| 2.0 | 43.3 ± 0.05 | 65.7 ± 0.2 | 69.4 ± 0.4 | 102.8 ± 0.2 | 180.5 ± 10.8 | 244.7 ± 4.2 |
| 2.5 | 42.1 ± 0.1 | - | - | - | - | - |
| 3.0 | 41.7 ± 0.2 | - | - | - | - | - |
| 3.5 | 40.6 ± 0.2 | - | - | - | - | - |
| 4.0 | 40.5 ± 0.1 | 40.4 ± 0.1 | 40.5 ± 0 | 40.5 ± 0 | 40.3 ± 0.05 | 40.3 ± 0.05 |
| 5.0 | 34.1 ± 0.05 | - | - | - | - | - |
| 6.0 | 29.2 ± 0 | - | - | - | - | - |
| NVIDIA IQ2 GGUF | 27.8 ± 0.1 | 27.2 ± 0.4 | 27.6 ± 0.3 | 27.4 ± 0.5 | 27.6 ± 0.05 | 27.5 ± 0.1 |
Against NVIDIA's 2-bit GGUF
NVIDIA's own small release is an IQ2_XXS GGUF made with quantization-aware distillation, 22.0 GB. Our 2.0 pack is about the same size, and we converted it from the BF16 checkpoint with no extra training. On the same 8,192 positions our pack is closer to BF16 on every measure we took, and it decodes faster than NVIDIA's file in both engines (43.3 ± 0.05 against 27.8 ± 0.1 and 38.2 ± 0.2 tok/s). We ran NVIDIA's file in TandemLLM, which reads the GGUF as stored, and in llama.cpp for context: commit 620d2225c with NVIDIA's flags for the Spark, except a 32,768-token context and no image projector, and --spec-type draft-dflash with the LiLiCorr GGUF and 7 draft tokens.
| EXL3 2.0 | NVIDIA IQ2 GGUF | |
|---|---|---|
| text weights, GB | 21.7 | 22.0 |
| KL vs BF16 | 0.170 | 0.259 |
| top-1 vs BF16 | 84.0 % | 79.6 % |
| perplexity (BF16 5.97) | 6.81 | 7.48 |
| plain decode, tok/s | 43.3 ± 0.05 | 27.8 ± 0.1 (llama.cpp: 38.2 ± 0.2) |
| LiLiCorr, prose ask, tok/s | 65.7 ± 0.2 | 27.2 ± 0.4 (llama.cpp: 49.3 ± 0.2) |
| LiLiCorr, chat ask, tok/s | 69.4 ± 0.4 | 27.6 ± 0.3 (llama.cpp: 62.0 ± 1.0) |
| LiLiCorr, code ask, tok/s | 102.8 ± 0.2 | 27.4 ± 0.5 (llama.cpp: 92.2 ± 1.0) |
| LiLiCorr, edit ask, tok/s | 180.5 ± 10.8 | 27.6 ± 0.05 (llama.cpp: 92.5 ± 1.4) |
| LiLiCorr, copy ask, tok/s | 244.7 ± 4.2 | 27.5 ± 0.1 (llama.cpp: 132.0 ± 0.9) |
llama.cpp with LiLiCorr gave a different text from its own plain run on the prose and chat asks, so its numbers there are not for the same output. TandemLLM's text with LiLiCorr was the same as its plain text on every ask, with both files. On NVIDIA's file, TandemLLM 0.10.0 decodes plain at 27.8 ± 0.1 tok/s against llama.cpp's 38.2 ± 0.2. A faster IQ2 kernel we measured at 32.7 tok/s is not in this release.
An all-2-bit pack, with the Mamba2, attention, latent and shared projections at 2 bits too, scored KL 0.379 and top-1 73.9 % on the same rows, worse than NVIDIA's file. Keeping those projections at 4 bits adds about 1.7 GB, and that is the pack in this repository.
How to run it
TandemLLM v0.10.0 runs every level (release notes). On a DGX Spark, install it in setup mode, then switch to this level:
curl -fsSLO https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.10.0/install.sh
bash install.sh --setup --yes
~/TandemLLM/run/ops/use-profile.sh nemotron-bolt-exl3-2.0
use-profile.sh downloads this repository at a pinned commit, checks each file against its sha256, fetches NVIDIA's LiLiCorr drafter and starts the engine on port 8000. It needs no Hugging Face token. A start needs about 20 GiB for the weights. All seven levels are in the collection.
The files are in the EXL3 format, but stock exllamav3 1.6.2 does not know Bolt's layer layout and will not load them. exllamav3 1.6.2 with the patch in convert/ loads them; we measured them with it. We have not tested other EXL3 loaders.
How we made it
convert/exllamav3-1.6.2-bolt.patch applies to exllamav3 v1.6.2 (commit 4c84273). It reads Bolt's per-layer block table and its Omni wrapper, gives each MoE layer its own expert count, and copies the image tower unquantized. convert/README.md has the commands. Each pack used exllamav3's built-in calibration (250 rows of 2,048 tokens) and the mul1 codebook; quality/convert.txt is the log.
License
These weights are a quantization of NVIDIA's model and carry its license, the OpenMDW License Agreement 1.1, in LICENSE. NOTICE says where the files come from and what we changed. The patch in convert/ changes exllamav3, which is MIT-licensed. Everything else about the model, including its intended use and limits, is in NVIDIA's model card.
Credits
NVIDIA made Nemotron 3.5 Bolt, its BF16 checkpoint, the IQ2 GGUF and the LiLiCorr drafter (NVFP4, GGUF). turboderp made exllamav3 and the EXL3 format.