The checkpoint · Speed · Quality · What it fits on · Use it · The base model · Bench card

Nemotron-3.5-Super-VL-120B-A12B-W4A16-Merlin

A 4-bit build of NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, NVIDIA's 120B mixture-of-experts vision-language model (12B active per token; Mamba-2, attention and MoE layers with a C-RADIO vision encoder), made to run at its full 1,048,576-token (1M) context on four 24 GB GPUs. It is 68 GiB instead of the original 232 GiB: the routed experts are INT4, the Mamba-2 projections INT8, and the main model's attention and shared experts and the vision tower stay in BF16 (the MTP head's are INT8). It is served by the Merlin engine, a vLLM 0.32.0 fork tuned for Ampere and Ada that runs it across four cards with expert parallelism, Mamba-2 RecoverSSM and MTP speculative decoding. Measured on RTX 3090s; the engine also ships profiles for other Ampere and Ada cards, from RTX 4090s and 48 GB workstation cards to A100s and CMP 170HX (see What it fits on). This is an independent derivative made by halt95, not made, reviewed or endorsed by NVIDIA; NVIDIA's own model card is kept unchanged as NVIDIA-MODEL-CARD.md (their revision 1f8953f7, 2026-10-16).

_{numbers: Merlin 3.0.1 on 4× RTX 3090 w/o NVLINK @ 220W, P2P on, with the optional Zagitta's gpu-unlock host module, 2026-11-02 bench card (runs of 2026-11-01 to 2026-11-02). KV pool: FP8, all in VRAM, at the shipped 1M context (the 3.0.0 card's 1,392,275 tokens was measured at a 262K context length); one 1M or five 262K sessions held at once (3.0.0 run, 2026-10-20). Agents: eight 16K conversations, then eight simultaneous 512-token follow-ups, thinking off. Decode: one stream, thinking on (this template has no effort levels, so it is full thinking), exactly 261,888 prompt tokens + 128 out, median of 3. Prefill: one cold 106,725-token prompt. Each figure is also in a table below.}

The checkpoint

_{Formats from the quantization_config in config.json, also listed in NOTICE; the KV cache format is set by the engine profile.}

Files in this repository

_{About 68 GiB in all. Also in the repository: a 136-byte model-summary-idxs.safetensors (NVIDIA's two-element summary_idxs tensor for the vision encoder, unchanged, re-saved with a format: pt header; NVIDIA's file is 104 bytes), LICENSE (OpenMDW-1.1), NOTICE (every file we changed), SHA256SUMS, NVIDIA's model card (NVIDIA-MODEL-CARD.md, revision 1f8953f7, unchanged) and its bias.md, explainability.md, privacy.md and safety.md and two charts. OpenMDW-1.1 requires anyone redistributing the checkpoint to keep a copy of the licence and NVIDIA's copyright and origin notices. The weights were built from NVIDIA's revision 533d6623.}

Speed

_{4× RTX 3090, the needle-prompt ladder (4,269 / 34,147 / 136,625 / 261,888 prompt tokens, counted by Nemotron's tokenizer), every send a cold prefill except at 262K, where send 1 is cold and sends 2 and 3 reuse the prefix before the needle. Decode: thinking on (this template has no effort levels, so full thinking), ≤ 256 tokens out (128 at 262K), median of 3. Time to first token: the median below 262K; the cold send 1 at 262K. The 262K needle is found in 3 of 3 thinking-off sends (3.0.0 run, 2026-10-20).}

_{One cold request per prompt, 8 tokens out at 10K and 100K. The 261,888-token row is the 2026-11-01 run's cold thinking-on send at 262K (128 tokens out), so it can differ slightly from the exact-262K figures above.}

_{N 16K conversations, then N simultaneous 512-token follow-ups, thinking off unless stated, total tok/s while all N decode. The 2- and 4-agent rows were not re-run. With thinking on a 512-token follow-up may end inside the reasoning, and reasoning tokens count, so that total measures decoded tokens, not finished answers. Each overlap window was short (7.34 and 8.61 s for 2 and 4 agents; 8.89 s thinking off and 8.63 s thinking on for 8), so read these rows loosely.}

_{N needle prompts (thinking off, greedy) sent at once, then re-sent one by one: every re-send served from the prefix cache means the pool held all N. Peak pool use 95.0 %, 0 preemptions. Nine is the engine's sequence limit for this checkpoint, not the pool's. 3.0.0 run (2026-10-20), not re-run for 3.0.1.}

Against the previous release (3.0.0)

_{tok/s. Decode: median of 3, the same frozen driver; thinking on is full thinking on both cards. Prefill: one cold send. Not like for like: the 3.0.1 runs used the shipped 1M context (the 3.0.0 pool was measured at a 262K context length) and the optional Zagitta's gpu-unlock host module (the 3.0.0 card does not list it), and no stock-versus-unlocked A/B was run for this checkpoint, so the prefill gain is not split between the unlock and anything else; the 3.0.1 prompts at 10K and 100K are longer (token counts in the rows); the 3.0.0 agents figure is from a pre-release engine build of 2026-10-17. Cold time to first token at 262K: 66.9 s against 88.8 s. Thinking-off decode at 262K is 1.2 % lower than in 3.0.0, over a 128-token sample. GSM8K (thinking on now, off on the 3.0.0 card) and the KV pool (1,392,275 tokens at a 262K context length in 3.0.0, 1,856,367 at the shipped 1M context) are not compared.}

How it was measured

• Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02; the 3.0.1 rows also with the optional Zagitta's gpu-unlock host module.
• Engine: Merlin 3.0.1 (its engine wheel is unchanged from 3.0.0), served by merlin serve with the shipped defaults, the 1M context included, one fresh boot per layout. The 3.0.1 rows were measured 2026-11-01 to 2026-11-02 on release candidates just before the final 3.0.1 commit, not on the final commit; rows marked 3.0.0 run were not re-run. Decode and prefill used prompts up to 262K; speed at a 1M-token prompt has not been measured.
• Layout: TP2 × PP2 + EP on four cards (layer split 46 / 42), Mamba-2 RecoverSSM, MTP with 3 draft tokens, FP8 KV, thinking on. This template has no effort levels, so thinking on is its full thinking; the bench driver's reasoning_effort medium has no effect.
• Decode: T=0, a needle prompt at depth, every send a cold prefill except at 262K, where send 1 is cold and sends 2 and 3 reuse the prefix before the needle; one discarded warm-up, then 3 sends per cell, median. tok/s = tokens after the first streamed event / time from first to last event.
• The ladder's prompts are sized in Qwen tokens and Nemotron's tokenizer counts the same text about 4.8 % longer, so the ladder rows above are Nemotron's own counts and the 262K rows are the exact 261,888-token sends. The two single cold sends were scaled to match the Qwen checkpoints' 10K and 100K prompts in the 3.0.0 run (10,187 and 101,885 tokens); the 3.0.1 sends are 10,671 and 106,725 tokens.
• Streams are client-timed. Full tables, thinking-off rows and ITL: the 3.0.1 bench card; the previous release: the 3.0.0 bench card.

Quality

This checkpoint, measured on the Merlin engine:

_{GSM8K with thinking on: Merlin 3.0.1, 2026-11-02, the 3.0.0 set (the same 200 problems, 5-shot, same scorer, temperature 0) on four cards. Rows marked 3.0.0 run were not re-run for 3.0.1: GSM8K with thinking off and the 131K and 262K needle panel come from the pre-release engine build of 2026-10-17; the exact-262K needle and the image from the Merlin 3.0.0 release wheel, 2026-10-20; the 1M needle from Merlin 3.0.0 at its 1M default (max_model_len 1,048,576), 2026-10-21. The two GSM8K scores differ in setting (thinking on against off), so they are not a before-and-after pair. These are checks of this checkpoint, not a matched BF16 comparison: no BF16 run of the same checks was made, so how much quality the 4-bit build keeps is not quantified here.}

The base model, as reported by NVIDIA for the BF16 weights:

_{A selection from the benchmark table on the upstream card (its BF16 column, headed "Nemotron 3.5 SuperGA BF16"), which has the full table and comparisons with other models. These were not re-run on this 4-bit build.}

What it fits on

_{Measured on the reference host (3.0.0 run, 2026-10-20 bench card; not re-run for 3.0.1): five 262K and nine 131K sessions held at once with every re-send served from cache; a 4096 × 2048 image with at least 686 MiB free on every card. The KV pin (2.50 GiB per card) has not had a sustained-load test yet. At the 1M default (3.0.0 run, 2026-10-21 and 22): a 1,048,052-token needle found exactly and, re-sent, served from the resident cache (1,047,936 tokens cached, 3 s to first token, no preemptions); five 262K and nine 131K needle prompts sent at once all answered; the 8 MP image left at least 666 MiB free on every card. The KV pool is 1,856,367 tokens for 2.50 GiB per card at the shipped 1M context, the figure on the 3.0.1 bench card; the 3.0.0 bench card's 1,392,275 tokens was measured at a 262K context length. The sessions above are what was measured.}

Other GPUs. The engine ships profiles for these, all on four cards; the Status column says how far each has been run:

_{merlin serve and ./setup.sh pick the profile from the GPUs they find. A ~ is the engine's estimate, not a measurement. Context fit is what the KV pool holds; the server admits at most 9 requests at once on 4× RTX 3090, RTX 3090 Ti and RTX A5000 with P2P (svl-quad-3090), 10 on the other rows with a 5 × 262K pool (including 4× RTX 3090 without P2P), and 32 on the larger-pool rows (each profile's sequence limit); the rest queue. Every profile uses four cards and an FP8 KV cache, on sm_80 cards too (FP8 KV ran on the 4× CMP 170HX 64 GB rental, needles exact). 🟢 measured by us · 🟡 testing: full-model runs passed on rented, cloud-hosted cards, kernel parity qualification still to run · ⚫ a profile ships, not yet qualified by us (expected to work; a rented-host run is noted in the row where one was made). Every profile: the engine's hardware matrix.}

_{Host requirements are the engine's; disk is this repository's size.}

Use it

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
./setup.sh --model nemotron-super-vl

One command does the rest: after a small uv and Python bootstrap it checks the GPUs, driver, RAM and disk, shows the engine, package and checkpoint downloads with their sizes before fetching them, installs the prebuilt engine, downloads and verifies this checkpoint, and starts an OpenAI-compatible server on 127.0.0.1:8000. Enter takes the recommended answer at each question; --yes takes them all, --dry-run shows the plan. Run ./setup.sh again to start the server, and git pull && ./setup.sh to update the engine without downloading the model again. Requests use the model name svl.

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

Manual install

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
curl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl
MERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" \
  bash release/install-env.sh ./venv

venv/bin/hf download halt95/Nemotron-3.5-Super-VL-120B-A12B-W4A16-Merlin --revision a0c5c92c01ac87bee8fd94299897970bc83072bf \
  --local-dir ~/models/nemotron-super-vl
(cd ~/models/nemotron-super-vl && sha256sum -c SHA256SUMS)
venv/bin/python merlin_cli.py serve ~/models/nemotron-super-vl

MERLIN_WHEEL installs the prebuilt engine (sm_80, sm_86, sm_89), checked against release/merlin-wheel.sha256; without it the installer compiles the engine instead, about 1.5 hours on 8 cores. merlin serve recognises the checkpoint, picks the largest tested layout for the GPUs it finds (or, with none, the largest untested one that fits, labelled as such) and serves on 127.0.0.1:8000. Add --dry-run to print the resolved command without starting anything. The first start compiles kernels.

Recommended settings (NVIDIA's, for this model):

_{NVIDIA's recommendation on the upstream card. Reasoning tokens and the answer share the max_tokens budget.}

• Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
• Reasoning levels are NVIDIA's own: off, on, and on (medium) with "chat_template_kwargs": {"enable_thinking": true, "medium_effort": true}, which writes fewer reasoning tokens and, NVIDIA notes, may lower accuracy. Merlin serves full thinking by default, and every thinking-on figure on this card is full thinking; reasoning_effort (the Qwen checkpoints' switch) has no effect on this model, whose template has no such effort levels.
• Chat template: Merlin serves the checkpoint's own Nemotron template, with the nemotron_v3 reasoning parser and the qwen3_coder tool-call parser, the way NVIDIA's card serves it. The Froggeric template the Qwen checkpoints use does not apply.
• One image per prompt, downsized by the model's processor to at most 13,312 patches (about 3.4 MP); images up to the engine's admission limit of 8,388,608 px are accepted. Video is off in Merlin's launcher.

The base model

Nemotron 3.5 Super VL is NVIDIA's vision-language model for reasoning, coding, tool use, documents and video. The architecture, in short:

• Hybrid layers. Mamba-2 state-space layers, Mixture-of-Experts layers and a few attention layers, interleaved: the Mamba-2 layers keep long-context cost down.
• Latent experts. 512 routed experts per MoE layer, 22 active plus 1 shared per token, run through a 1,024-wide latent projection: 12B of the 120B parameters do the work for each token.
• Vision. A C-RADIO encoder and a vision-language connector take images, document pages and video frames.
• Multi-token prediction. A shared MTP head trained for speculative decoding; Merlin uses it with 3 draft tokens.
• Reasoning levels set in the chat template: off, on, and on (medium).

_{Architecture as reported by NVIDIA on the upstream card; layer and head counts from its config.json.}

Other builds

_{One stream, thinking on, 261,888 prompt tokens + 128 out, median of 3; Merlin 3.0.1, 2026-11-02 bench cards. Thinking on at reasoning effort medium, the server default, wherever the checkpoint has effort levels; the Nemotron and Ornith 1.5 templates have none, so those two ran at full thinking.}

Good to know

• Every figure we measured for this checkpoint comes from 4× RTX 3090 at 220 W, PCIe Gen4 with P2P (the 3.0.1 speed rows also with the optional Zagitta's gpu-unlock host module), except the rental rows under Other GPUs; the Other builds table names each build's card count. Hosts without P2P use the no-P2P profiles (see What it fits on).
• Four cards are the minimum: two-card hosts are refused.
• Independent derivative: not made, reviewed or endorsed by NVIDIA. NVIDIA's model card (NVIDIA-MODEL-CARD.md, revision 1f8953f7) covers training data, intended use, evaluation and limitations; read it before use.

Changes

_{Built on the reference host; the Hub repository starts with this build. Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.}

Credits and licence

NVIDIA (model; OpenMDW-1.1, see LICENSE; NVIDIA's model card kept unchanged as NVIDIA-MODEL-CARD.md), the vLLM project (the engine Merlin is built on), aikitoria (peer-to-peer on consumer Ampere), Neiko2002, and CoolShades (the idle-power report behind the v2.5.2 engine fix). Upstream fixes are credited in the engine repository.