NVIDIA Nemotron 3.5 Bolt 74B-A9B EXL3 2.08 bpw
EXL3 quantization of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16, sized to run text inference on one RTX 3090 24GB.
That BF16 card names nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 as its own base model. Super VL is lineage. It is not this repo's base_model.
This repository is the language pack only. It does not serve images or video.
Quant
| Total / active parameters | 74B / 9B |
| Architecture in this engine | NemotronHForCausalLM (Mamba-2 / Transformer hybrid LatentMoE) |
| Codebook | mul1 |
| Average bits | 2.08 |
| Routed experts | 2 bpw |
| Shared experts and Mamba projections | 3 bpw |
| Attention and latent projections | 4 bpw |
| Embeddings and lm_head | 6 bpw |
| Final norm | stored at 16 bpw, not a matrix |
| Nominal config context | 262144 |
| Served max_model_len | 262080 |
| Cache | 262144 tokens, 8-bit KV, one sequence slot |
| Engine | exllamav3 21ce2d1a82cab53c9e1301aef92f1239459278b8 (v1.6.1-7-g21ce2d1) |
| License | OpenMDW-1.1 |
served max_model_len is 262080 because a 262016-token prompt plus a 64-token generation reserve is the longest request this 1024-page cache completed. The config field remains 262144. A prompt of 262143 tokens plus one new token does not fit.
Converter reconstruction over 91 modules: mean rfn 0.053, mean sqnr 27.1. Worst module language_model.backbone.layers.48, rfn 0.147, sqnr 16.9, 2.04 bpw. That is not token KL. Token KL against the 148GB BF16 checkpoint was not measured.
Measured on one RTX 3090, no drafter
Prefill, raw /v1/completions, max_tokens=1, distinct token strides, same host, no shared prefix:
| prompt tokens | tok/s |
|---|---|
| 3072 | 2035.7 |
| 9216 | 2283.3 |
| 32768 | 2297.4 |
| 65536 | 2156.4 |
| 131072 | 1904.2 |
| 262016 | 1556.9 |
Decode:
| setup | tok/s |
|---|---|
| 128 tokens after an 8-token warmup, cache 8192, 8-bit KV | 80.4 |
| 128 tokens after a reused 32768-token prefix | 68.7 |
The cold 32768-token request that includes prefill is not a decode rate. Reserved memory at the 262144 cache was 22010 MiB. The prefill ladder held 21812 MiB.
Prefill-ladder telemetry, load-only: power mean 338.6 W / max 349.6 W, temp mean 61.9 C / max 65.0 C, util mean 95.9%, graphics clock mean 1759 MHz, no hardware thermal throttle (0x4 is the software power cap).
Q200v2
Kit status is INCOMPLETE. Eight IFEval rows hit the 8192 cap (finish_reason: length) and the frozen runner leaves them ungraded. The cap was not raised. There is no composite score.
Thinking was enabled. The request also sent thinking=true and reasoning_effort=low. This chat template reads only enable_thinking. Workers 1. Greedy. Same host. Sandbox image sha256:c01743fb16942c954251ade685bfb44378a978b64ff2f06b2444a59ed5153e06. Engine 21ce2d1a82cab53c9e1301aef92f1239459278b8. No drafter.
| family | correct | n | note |
|---|---|---|---|
| gsm8k | 76 | 80 | 95.0% @80 |
| humaneval | 38 | 40 | 95.0% @40, exec-graded in the sandbox |
| ifeval | 32 | 32 transported | 8 of 40 rows length-capped and ungraded. Not 32/40. |
| hard_reasoning | 19 | 20 | independent manual review. The miss is hard-12: the finite-graph proof is valid, and the unqualified claim is false for an infinite path. |
End-to-end throughput on these 180 rows, completion_tokens / elapsed_seconds, including the eight length rows: mean 79.37, p50 79.59, aggregate 80.03 tok/s, 217871 completion tokens. Per family mean: gsm8k 78.89, humaneval 79.19, ifeval 80.04, hard_reasoning 80.33.
Quality-run telemetry, load-only: power mean 345.4 W / max 349.4 W, temp mean 58.9 C / max 64.0 C, util mean 97.8%, graphics clock mean 1888 MHz, VRAM max 21816 MiB. Throttle values were idle and 0x4 software power cap. No hardware thermal throttle.
NIAH
Served max_model_len is 262080. A 128-token cap on a 65250-token prompt stopped on length before the code was emitted, so the recorded depths use a 1024-token cap and a 261000-token full window. That is a disclosed deviation from the 64-token protocol reserve.
| depth | prompt tokens | exact match | note |
|---|---|---|---|
| 25% | 65250 | no | code was retrieved as R0B0-Zebra-7731, not R0B0-ZEBRA-7731 |
| 50% | 130500 | yes | |
| 90% | 234900 | yes | |
| 100% | 261000 | yes | elapsed 117.9 s may include prefix reuse from the 90% prompt |
| last of 33/66 | 261000 | yes | expected R0B0-LYNX-4402 |
| last of 33/66/90 | 261000 | yes | expected R0B0-RAVEN-9158. Elapsed 28.6 s is prefix reuse, not a prefill measurement. |
Not run
- Speculative decoding. The paired LiLiCorr EXL3 pack does not load on this engine (
LiLiCorrDraftModelis unknown). No acceptance length. - Vision. A side pack of 132 RADIO and projector linears at 6 bpw exists and is not in this repository. This engine has no RADIO or Nemotron omni class.
- Streaming latency and inter-token latency. The serve returns 400 on
stream: true. - Concurrency above one slot. This cache has one sequence slot.
- BFCL. A tool-bearing chat request returned no
tool_calls. Not implemented, not a zero.
Usage
python serve_openai.py \
--target NVIDIA-Nemotron-3.5-Bolt-74B-A9B-EXL3-2.08bpw \
--host 127.0.0.1 --port 8891 \
--max-model-len 262080 --cache-tokens 262144 --cq 8
Do not pass a draft directory. The base model's recommended sampling is temperature 1.0, top_p 0.95. The measurements above are greedy.