Nemotron-3.5-Super-120B-A12B — NVFP4 + Strata engine for one RTX 50-series card

186 t/s decode and 9,610 t/s prefill at 32K on one RTX 5090 with the experts in RAM (130 t/s in the precise decode mode): 6× llama.cpp's decode on the same weights, with NVIDIA's own NVFP4 experts and the MTP head kept.

Blackwell only: RTX 50-series (sm_120). Every optimization of the engine is built on Blackwell's FP4 tensor-core instructions (mma … kind::mxf4nvf4.block_scale, NVFP4 × NVFP4 with its native scales), which no earlier generation has. The engine is compiled for sm_120a and does not run on RTX 40/30-series or on AMD.

Vision works. Images go through strata-vision with Unsloth's mmproj-BF16.gguf and a patched llama.cpp (see Images below). Its embeddings match NVIDIA's own code to a per-token cosine of ≥ 0.998. Video is not done yet.

16 GB cards work (5080, 5070 Ti, 5060 Ti 16G; with 128 GB of RAM). With MTP and the precise decode: 58–93 t/s decode, 2,400–4,600 t/s prefill, decode PPL ×1.010. Measured by emulation on the 5090 (--vram-limit-mib 15800), see Hardware configurations. 64 GB of RAM is in development: cold experts streamed from an NVMe SSD.

This repo has a GGUF of the text model of NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4, made for strata-nemotron. That is a fork of the Strata engine that runs this model on one consumer card: 63 GB of experts in pinned RAM, a VRAM cache in front of them, and every NVFP4 GEMM of the prompt on Blackwell's FP4 tensor cores.

The routed experts are NVIDIA's NVFP4 values, bit for bit, repacked to ggml's NVFP4 without loss. The 1.6 GB multi-token-prediction (MTP) block is kept: it takes decode at 32K from 109 to 186 t/s (fast mode) and from 52 to 130 t/s (precise). The vision tower is not in this file: it is Unsloth's mmproj (see Images).

Performance

strata-nemotron has two decode modes. They differ only in what happens to a cache miss whose routing weight is low:

• fast (--skip-w 0.4 --prefetch 0): the miss is skipped;
• precise (--skip-w 0.4 --prefetch 1): the next layer's likely experts are prefetched, so far fewer misses are skipped.

Prefill is the same in both.

* llama.cpp cannot load NVFP4 inside the MTP block, so it ran an MTP-free variant of this file (convert.py --no-mtp) with --fit on.

Greedy, 256 generated tokens on wikitext-2 (test + train for the 256K–1M rows; "1M" is 1,047,552 prompt tokens, the window minus 1K), one process per row, so the expert cache starts cold. At 1M the K/V (~6.5 GiB) has taken 56 slots a layer from the expert cache, which is why MTP matters most there: 2.4× decode (fast), 2.6× (precise). MTP is 5 drafts with a 0.8 probability cut-off, and the "tokens a window" figures in the precise mode are 2.23 / 3.56 / 2.05 / 2.56 / 2.21 / 3.76 at 4K / 32K / 128K / 256K / 512K / 1M. The decode PPL is --ppl-decode on wikitext-2 (4 × 1024), against the same engine with no skipping.

Real use, not a benchmark: all 363 requests the v3 engine served through llama-swap in the fast decode mode, on the machine of Tested hardware (everyday chat plus the checks in this card: tool calls, images, the 512K–1M needle tests), with MTP and the sampling the clients asked for. Generation is 181 · 229 · 247 t/s (p50 · p95 · p99) and never below 106 t/s. Prompt is 1,257 t/s at p50, because most requests are short follow-ups whose prompt is a few hundred tokens, and 9,169 t/s at p95.

The fastest prompts in the histogram, through the server:

• 15.6K tokens: 11,397 t/s;
• 62K tokens: 10,499 t/s;
• 1M tokens: ~3,050 t/s (340 s).

The engine alone (--bench, the second prompt in a row, so the expert cache is warm) is ~10% faster: 12,700 t/s at 16K, 11,270 at 64K, 7,440 at 256K. Through the server, a prompt is read in two parts: up to the last complete message, where the state is saved for the next turn, and then the rest. The cache is also not always warm there.

Needle in a haystack at 32K, 128K, 256K, 512K and 1M (depths 10/50/90%) found 15/15.

Up to ~8K, a prompt is slower than you would expect from the 32K row. Top-22 of 512 experts touches almost every expert of every layer even on a short prompt, so ~40 GB has to cross PCIe whatever the length. From 16K on, compute dominates.

Update 2026-10-22: agentic sessions

The tables above are for one long prompt and a long answer. An agent session looks different: a prefix of 60K+ tokens, then many turns of a few hundred new tokens each, sometimes with a request from another conversation in the middle, an edited earlier message, or a screenshot. This update is about those. Decode speed did not change: the decode rows above stand, and in the before/after runs below the difference was −2.6% at 6K, −1.9% at 32K and +0.9% at 32K with temperature 1.0, against an 8–12% spread between identical runs.

Time to first token, before and after, through the server (RTX 5090 of Tested hardware; temperature 0; one server run per arm, the arms ~15 minutes apart; short turns are the median of 2 repeats):

"Before" is the engine of the rest of this card (--prompt-skip-w 0.3, one saved state). "After" adds the options below. In the "after" run five experimental decode switches (environment variables, off by default, not part of the published setup) were also on; they showed no effect on decode, but the short-turn gain was not measured without them. The prefix of 75,474 tokens is generated text; "cold" pictures are pixel-edited copies, so each is encoded again.

What the conversation cache does. A Mamba state cannot be rolled back, so the engine can continue a conversation only from a position whose state it saved. It used to save one state, at the last message boundary, in VRAM. Every turn therefore read the previous answer again (about 45 tokens of template and answer), and any other request, such as a title generator or a sub-agent, replaced the saved state, so the next turn read the whole 81K tokens again (7.8 s). Now the engine keeps up to N states in pinned RAM: at message boundaries, every 16K tokens of a long prompt, and right after each answer, so the next turn's prompt matches a saved position exactly.

When another conversation arrives, the current one is parked: its K/V cells and its deepest saved states are copied to pinned RAM (60K tokens: 21–43 ms and 729–907 MiB), and copied back when a request extends it (60K: 17–18 ms). A branch from an earlier turn also finds its state. Resuming is bit-identical to the old single state (logit difference 0.0 on the same tokens). It costs RAM and a few ms on a warm request: the prompt step went from 31–35 ms to 41–49 ms. Limits: when the answer is re-rendered differently from what was generated, the post-answer state does not match and each turn can park a copy, which the budget evicts oldest first; and a resume equals a cold read bit for bit only when the prompt is split into the same chunks. Details: docs/nemotron/CONVERSATION_CACHE.md.

New options (all off unless given):

The cache adds 8 × ~165 MB + 3000 MiB of page-locked RAM to the 59 GiB of experts.

--prompt-skip-w 0.3 → 0.4. Short turns are bound by copying experts over PCIe, so each expert not copied saves time. Measured with the wikitext-2 test set (16 chunks of 1,024 tokens) for PPL, and the median of 3 short turns after a 75K prefix for time (one server run per value; RAM and part of the GPU were shared with other jobs, so read the milliseconds as indicative):

The rule: a change is taken when the time falls by a larger percentage than the PPL rises. 0.4 against 0.3 is −44…−45% of the time on 30–400 token turns (−27% at 2,000) for +0.8% PPL. 0.5 and 0.6 pass the same rule but cost ×1.088 and ×1.361 here, so 0.4 stays. The PPL is measured on 1,024-token chunks, which is where this skipping acts; it is not a quality measure of a 75K-token session. Greedy texts of the old and the new build differ from each other no more than two runs of the same build do (the expert cache depends on history), and all of them were valid summaries of the same document.

Pictures. Two changes in the encoder's CPU part, both exact on the pixels. The resize now runs on 14 threads (half the hardware threads of the test machine) and skips the axis whose length does not change (the .sve embedding files are byte-identical to the old build on the 5 test pictures): the resize of a 1920×1080 picture went from 41 to 20 ms, a 2560×1440 one from 81 to 38 ms, and the whole encoder call (decode, resize, GPU encode, write) from 240 to 221 ms and from 469 to 424 ms. And WebP, TIFF and EXIF-rotated JPEG are now handed to the encoder as BMP instead of PNG (Pillow's PNG encode of a 1024×768 picture is about 13 ms; BMP takes 2–10 ms). The rest of the picture gain in the table is the same checkpoint effect as for text, plus --prompt-skip-w; the shares were not measured apart. A repeated picture is still encoded again on the next turn. Inside the server the encoder is slower than alone (246–431 ms at 1920×1080) because it shares the GPU with the engine.

What was tried and left out, with numbers: docs/nemotron/REVIEW_2026-10-22.md. Among them: reading each expert once per verify window (same tokens, but slower: 3× the kernel launches), and a Q8_0 vision projector (−786 MiB of VRAM and 4–11% faster, but on 4 of 5 pictures the worst per-token cosine was 0.947–0.997, under the limit of 0.998).

Quality

How these were measured:

• llama-perplexity, wikitext-2 test, 16 × 1024 tokens;
• the experts are the same NVFP4 in every row, and only the dense FP8 layers differ;
• the baseline row holds the checkpoint's own values, so the ratios are against NVIDIA's release, not against BF16 training weights.

These dense layers are ~6% of the weights, yet putting all of them in NVFP4 costs about three times the KL of NVFP4 on all the experts. lm_head (+1.3% PPL) and Mamba in_proj (+0.9%) are the expensive ones. This file takes the middle row: 2 GiB less VRAM than Q8_0 everywhere, which is ~700 more experts in the cache.

strata-nemotron's prompt path adds about ×1.008 PPL on top of this: ×1.003 for FP4 activations, ×1.004–1.005 for int8 attention. Short prompts (up to 8,192 rows) also skip low-weight cache misses (--prompt-skip-w): ×1.012 at 0.4 on 1,024-token chunks, see the update above. Decode adds ×1.008 in the precise mode and ×1.091 in the fast one (see Performance). Speculative greedy is bit-identical to plain greedy (300/300 tokens).

End to end, 25 prompts went through the OpenAI API (facts, arithmetic, code, SQL, bash, RU/EN translation, formulas; greedy). strata-nemotron (fast mode) and llama.cpp each got 25/25 correct, and 21 of the 25 answers were byte-identical. In the precise mode: 25/25 in two of three runs, 24/25 in one (a reasoning trace ran past the 1,024-token limit).

Parameters

How to run it

The file runs in strata-nemotron. It needs an RTX 50-series card (sm_120), and its prompt path is built for sm_120a. Build it:

git clone -b strata-nemotron https://github.com/Cudila/Strata strata-nemotron && cd strata-nemotron
cmake -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build -j --target strata-nemotron

The engine on its own:

./build/strata-nemotron --gguf Nemotron-3.5-Super-120B-A12B-NVFP4-00001-of-00002.gguf \
    --max-context 1048576 --kv-base 65536 --chunk 16384 --vram-reserve-mib 1024 \
    --spec 5 --spec-min-p 0.8 --skip-w 0.4 --tail-skip 0 --prefetch 0 --evict lfu \
    --prompt-skip-w 0.4 --prompt-skip-max 8192 \
    --prompt-cache 8 --conversation-cache-mib 3000 --serve

The last two options are the conversation cache (see the update 2026-10-22); leave them out and the engine keeps one saved state, as before.

With the OpenAI/Anthropic-compatible server (tool calls, reasoning, the chat template). Put both shards in models/, write the server's tokenizer folder from the GGUF once, then start it:

python tools/strata_tokenizer.py --gguf models/Nemotron-3.5-Super-120B-A12B-NVFP4-00001-of-00002.gguf --out models
python serve/server.py --engine strata --config docs/nemotron/strata-nemotron.example.json --port 8080

The first command writes models/tokenizer/: vocabulary, merges and the chat template, all taken from the GGUF. The example config uses paths relative to the repository. --gguf takes either shard: the other one is found next to it.

On startup the engine prints loaded in … s: dense 6.63 GiB in VRAM, 140 expert slots per MoE layer (16.1 GiB), K/V 65536 cells (grows to 1048576), experts page-locked in place. On a 32 GB card a slot count well below 140 means something else holds VRAM.

Images

The encoder is a separate process, strata-vision. It uses Unsloth's mmproj-BF16.gguf and is built against llama.cpp 098ce526b with the patch in the fork:

git clone https://github.com/ggml-org/llama.cpp ../llama.cpp && git -C ../llama.cpp checkout 098ce526b
git -C ../llama.cpp apply $PWD/tools/vision/patches/llama.cpp-nemotron-omni-cpe.patch
cmake -S tools/vision -B build-vision -DCMAKE_BUILD_TYPE=Release -DLLAMA_DIR=$PWD/../llama.cpp \
    -DSTRATA_VISION_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_PORTABLE=OFF
cmake --build build-vision -j --target strata-vision

With mmproj-BF16.gguf in models/, the example config's "vision" section starts the encoder. The server then takes OpenAI image_url parts and Anthropic image blocks.

Notes that cost time if you miss them:

• Give the desktop its VRAM. If the card also drives your screen, keep --vram-reserve-mib 1024 or more. A compositor or a screen-sharing portal that runs out of VRAM loses its buffers, and a remote-desktop session can lose its capture permission with them.
• --skip-w 0.4, --prefetch and --prompt-skip-w 0.4 are part of the speed. A cache miss whose routing weight is below the threshold is skipped instead of copied over PCIe. --prefetch 1 copies the next layer's likely experts ahead of time, so far fewer are skipped: precise decode, ×1.008 PPL. --prefetch 0 is the fast mode, ×1.091. Without skipping at all, decode runs at 43 t/s. --prompt-skip-w takes a 128-token prompt from 0.7 s to 85 ms (measured with 0.3).
• Loading takes 35–45 s. The 59 GiB of experts are copied into page-locked RAM once, from the page cache if the file is in it. Pageable memory would put every PCIe copy through the driver's staging at a fraction of the speed.
• The KV grows into the expert cache. Up to --kv-base (64K) it is fixed. Beyond that, each 64K page takes 4 slots of every layer from the cache: 140 slots at 64K, 128 at 256K, 84 at 1M. They are given back on the next conversation.
• Other MTP heads are not faster. --mtp-gguf loads ggml-org's separate mtp-*.gguf (BF16 / Q8_0 / Q4_0) instead of the file's own NVFP4 block. Their draft acceptance is the same within ±3% (2.14–2.20 tokens a window at 4K, greedy, identical main output), and the BF16 / Q8_0 ones take 32 / 8 expert slots a layer.
• llama.cpp does not load this file. It has no NVFP4 inside the MTP block. tools/nemotron/convert.py --no-mtp in the fork makes a variant it can load.

Tested hardware

Everything measured in this card was measured on exactly one machine:

Windows, AMD, other GPUs, multi-GPU and smaller RAM sizes were not tried.

Hardware configurations

What decides whether this runs is where its parts live:

The 24 GB row was emulated with --vram-reserve-mib on the same card, the 16 GB row with --vram-limit-mib. So it shows what a smaller cache costs, not what a laptop GPU's lower compute and power limit cost: read it as an upper bound.

In the fast mode a half-size cache is nearly free: the misses it adds are mostly experts whose weight is below the skip threshold, and long prompts are bound by compute. In the precise mode those misses are copied, so decode drops by ~27%.

RAM:

• The page-locked expert arena is 59 GiB and cannot be swapped.
• The process peaked at 105 GiB resident on the 128 GB machine, but that includes the file's pages in the page cache, which the kernel can drop.
• The conversation cache of the example config adds ~4.3 GB of page-locked RAM (8 × ~165 MB + 3000 MiB).
• 96 GB is probably enough, but was not tried. 64 GB is not: the arena alone does not fit next to the system.

Files

1189 tensors: BF16 88, F32 824, NVFP4 244, Q8_0 33. sha256: shard 1 98b3cd49…839be6, shard 2 60933aa7…b50ac3.

How it was made

1. Conversion. tools/nemotron/convert.py in the fork wraps llama.cpp's converter: --fp8 nvfp4 --nvfp4-only "mixer\.(in|out)_proj" --fp8-as-q8. On this checkpoint the stock converter needs four workarounds, which the wrapper applies without patching llama.cpp:
   • it drops the vision tensors;
   • it renames the MTP tensors before they are packed;
   • it reads n_routed_experts (otherwise all 59 GiB of experts pile up in RAM);
   • it treats quant_method: modelopt_mixed as ModelOpt for the FP8 dequantization.
2. FP8 → NVFP4 for the Mamba projections. It uses the same block-scale search as ModelOpt: the minimum-MSE neighbour among the E4M3 codes, not rounded amax/6. On the experts this encoder gives a relative error of 0.0812 against ModelOpt's 0.0813.
3. The choice of mix. The tensor classes were measured one by one in llama.cpp, as in the Quality table. The mix was then picked for speed at an acceptable perplexity.
4. Splitting. llama-gguf-split --split-max-size 45G. llama-gguf-split has to come from a llama.cpp build that knows the NVFP4 type (40). The split file gives the same perplexity in strata-nemotron as the single file: 3.2120 on 8 × 1024 tokens.
5. Checks. strata-nemotron-check checks every tensor against config.json: name, type, exact shape, file bounds. Every kernel of the engine has a parity test against a float64 transcription of the model's equations.

Caveats

• Images work, video does not yet. Stock llama.cpp's nemotron_omni stretches RADIO's position table on non-square pictures (per-token cosine down to 0.32 against NVIDIA's code). The fork ships a patch for it. strata-vision also uses NVIDIA's own tile size and resampling, which differ from mtmd's generic dynamic size.
• Blackwell (sm_120) only: the engine's kernels use FP4 tensor-core instructions that earlier GPUs do not have.
• One machine, one card (Tested hardware). The 24 GB row is an emulation; the smaller rows follow from the engine's own refusal message.
• The update of 2026-10-22 is one server run per arm on one machine: the gains in the table are far larger than the run-to-run spread we saw (8–12% in decode), but they are not confidence intervals.
• PPL ×1.016 against the checkpoint is a wikitext mean. No downstream benchmarks were run beyond the 25-prompt check.
• The reasoning modes, the chat template and the model's behaviour are NVIDIA's. This repo changes only how the weights are stored.

Credits

Engine, port and measurements: Cudila — https://github.com/Cudila/Strata (branch strata-nemotron).

The base model is by NVIDIA. The engine is a fork of Strata, and the conversion goes through llama.cpp. The weights are NVIDIA's and are governed by the OpenMDW License Agreement 1.1. NVIDIA's model card covers intended use, training data, bias and safety.