Nemotron 3.5 Super VL 120B-A12B in EXL3, 2.0 bits per weight
These are EXL3 weights of NVIDIA's Nemotron 3.5 Super VL 120B-A12B at 2.0 bits per weight: 2.11 bits on average over the quantized layers, 36.8 GB in all. We converted them from NVIDIA's BF16 checkpoint, nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 at revision 97e00975, with exllamav3 1.6.2 and a patch for the model's layer layout (in convert/). The routed experts are at 2 bits. The Mamba2, attention, latent and shared-expert projections are at 4 bits (exllamav3's -hq), and the output head is at 6. The MTP head keeps its experts at 4 bits and its other projections at 5 to 6. The embedding stays in BF16. The vision tower and its projectors are in the files unchanged, in BF16 (1.7 GB); TandemLLM serves text only for now and does not load them.
We serve these weights with TandemLLM, our inference engine for one DGX Spark. Super VL has 88 layers: 40 Mamba2, 40 mixture-of-experts with 512 experts each (22 a token, plus one shared expert) and 8 attention layers. One more layer forms the MTP head. The model reads 1,048,576 tokens of context; TandemLLM serves up to 262,144.
Which pack
This pack loses a lot. KL against BF16 is 0.325, more than six times the 4.0 pack's 0.049, and the top-1 token matches BF16 82.1 % of the time against 92.9 %. The same recipe on Nemotron 3.5 Bolt gave KL 0.170. It is 43 % smaller than 4.0 and 17 % faster plain. Use it only when the 4.0 pack does not fit; 4.0 is the one we recommend.
Quality against BF16
We ran each weight set in TandemLLM on 8 rows of wikitext-2, 1,024 tokens each (8,192 positions), and compared its logits with BF16's on the same tokens. KL is the mean KL divergence from BF16 a position. Top-1 is how often both pick the same next token. GB is the download.
| weights | GB | KL | top-1 | perplexity |
|---|---|---|---|---|
| EXL3 4.0 | 65.0 | 0.049 | 92.9 % | 3.94 |
| EXL3 2.0 | 36.8 | 0.325 | 82.1 % | 4.93 |
| NVIDIA NVFP4 | 75.1 | 0.041 | 93.4 % | 3.83 |
| BF16 | 248.9 | 0 | 100 % | 3.84 |
exllamav3's eval/model_diff.py on 100 rows of 2,048 tokens gives the 4.0 pack KL 0.0224, top-1 94.3 % and perplexity 4.726 (BF16 4.673); that run is in quality/ of the 4.0 repository. We did not run it on 2.0.
Speed on a DGX Spark
One DGX Spark (GB10, 128 GB), nothing else running. One request at a time, 512 greedy tokens, thinking off. Each TandemLLM run started a fresh server with a 140,000-token window: one slot without a drafter, three slots with NVIDIA's LiLiCorr drafter plus TandemLLM's lookup. StairCut picks the block size. Each set ran three plain and three drafted runs, alternating. Edit and copy are asks whose answer repeats much of the prompt. Each cell is tok/s, the mean ± half the range of the three runs.
For context we ran NVIDIA's NVFP4 release in vLLM (nightly-3f1bcfd8) with the flags from NVIDIA's card for the Spark, text only, three runs each: at most 4 sequences plain and 1 drafted, 75 % of the memory, the checkpoint's FP8 KV cache.
| weights | engine | drafter | prose | chat | code | edit | copy |
|---|---|---|---|---|---|---|---|
| EXL3 4.0 | TandemLLM | none | 31.3 ± 0.0 | 31.1 ± 0.0 | 31.2 ± 0.0 | 31.0 ± 0.0 | 31.0 ± 0.0 |
| EXL3 4.0 | TandemLLM | LiLiCorr + lookup | 33.1 ± 0.7 | 43.9 ± 0.7 | 53.6 ± 2.9 | 101.0 ± 3.3 | 108.3 ± 1.1 |
| EXL3 2.0 | TandemLLM | none | 36.7 ± 0.0 | 36.5 ± 0.1 | 36.5 ± 0.1 | 36.3 ± 0.1 | 36.3 ± 0.1 |
| EXL3 2.0 | TandemLLM | LiLiCorr + lookup | 35.6 ± 1.8 | 46.2 ± 0.6 | 46.3 ± 14.0 | 125.1 ± 1.6 | 139.2 ± 0.4 |
| NVIDIA NVFP4 | TandemLLM | none | 19.7 ± 0.0 | 19.6 ± 0.0 | 19.6 ± 0.0 | 19.5 ± 0.0 | 19.5 ± 0.0 |
| NVIDIA NVFP4 | TandemLLM | LiLiCorr + lookup | 23.9 ± 0.0 | 28.7 ± 0.3 | 39.9 ± 0.0 | 73.5 ± 0.2 | 81.7 ± 2.6 |
| NVIDIA NVFP4 | vLLM | none | 19.4 ± 0.1 | 19.4 ± 0.1 | 19.4 ± 0.0 | 19.4 ± 0.1 | 19.4 ± 0.0 |
| NVIDIA NVFP4 | vLLM | LiLiCorr, 7 tokens | 29.9 ± 0.9 | 34.8 ± 0.4 | 52.7 ± 2.0 | 72.4 ± 0.3 | 72.7 ± 0.3 |
| NVIDIA NVFP4 | vLLM | MTP head, 5 tokens | 29.1 ± 1.4 | 32.3 ± 2.1 | 43.2 ± 1.6 | 52.9 ± 0.1 | 53.1 ± 0.2 |
With LiLiCorr and the lookup, TandemLLM wrote the same text as its plain run on every ask, in every run. On 20 API checks (tools, JSON, thinking budgets) this pack gave the same answer drafted and plain on 17. On the other 3 a reasoning budget forced the thinking block closed, and the close landed a few tokens later. vLLM's greedy text changed from run to run on the prose, chat and code asks, plain runs included. NVIDIA's flags round the Mamba cache at random. Its speeds were steady, but its drafted text can't be checked against its plain text.
On the 2.0 pack the drafted prose and code runs vary: code went from 36.4 to 64.3 tok/s, and drafted prose is no faster than plain.
vLLM reads a prompt 4 to 5 times as fast: 2,158 to 2,521 tok/s against TandemLLM's 476 to 589, at 7,233 to 114,568 prompt tokens. With three requests at once vLLM batches them to 35.9 tok/s in total, plain. TandemLLM's drafted slots take turns: 19.6 tok/s in total on the 4.0 pack.
Size and memory
| EXL3 4.0 | EXL3 2.0 | NVIDIA NVFP4 | |
|---|---|---|---|
| download, GB | 65.0 | 36.8 | 75.1 |
| weights in memory, GiB | 58.4 | 31.2 | 66.6 |
| free memory a start needs, GiB | 90 | 64 | 95 |
The start figure covers the weights, three slots, LiLiCorr, the prefill workspace and a 15 GiB floor for the system. TandemLLM does not load the MTP head; with it the 4.0 pack takes 59.8 GiB.
How to run it
TandemLLM v0.11.0 runs both packs and NVIDIA's NVFP4 release (release notes). On a DGX Spark, install it in setup mode, then switch to this pack:
curl -fsSLO https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.11.0/install.sh
bash install.sh --setup --yes
~/TandemLLM/run/ops/use-profile.sh nemotron-super-vl-exl3-2.0
use-profile.sh downloads this repository at a pinned commit, checks each file against its sha256, fetches NVIDIA's LiLiCorr drafter and starts the engine on port 8000. It needs no Hugging Face token. The Setup Guide in TandemLLM's dashboard lists the same profiles. Both packs and NVIDIA's releases are in the collection.
The files are in the EXL3 format, but stock exllamav3 1.6.2 does not know Super VL's layer layout and will not load them. exllamav3 1.6.2 with the patch in convert/ loads them; we measured them with it. We have not tested other EXL3 loaders.
How we made it
convert/exllamav3-1.6.2-supervl.patch applies to exllamav3 v1.6.2 (commit 4c84273). It reads the per-layer block table, the Omni wrapper and the MTP head under it, and copies the vision tower unquantized. convert/README.md has the commands. Each pack used exllamav3's built-in calibration (250 rows of 2,048 tokens) and the mul1 codebook; quality/convert.txt is the log.
License
These weights are a quantization of NVIDIA's model and carry its license, the OpenMDW License Agreement 1.1, in LICENSE. NOTICE says where the files come from and what we changed. The patch in convert/ changes exllamav3, which is MIT-licensed. Everything else about the model, including its intended use and limits, is in NVIDIA's model card.
Credits
NVIDIA made Nemotron 3.5 Super VL, its BF16 checkpoint, the NVFP4 release and the LiLiCorr drafter. turboderp made exllamav3 and the EXL3 format.