NVIDIA-Nemotron-3.5-Super-VL-120B-A12B — APEX i-quality

APEX (structural per-tensor allocation) GGUF of nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 — 120B total / 12B active, hybrid Mamba2/attention/latent-MoE.

This repo is the text LLM only. The vision mmproj and the MTP head are separate files in the ggml-org source repo; they are not part of this quant.

Status

• i-quality is the only size built so far. i-compact / i-mini are planned as a size sweep. The BF16 source was deleted after this quant to reclaim disk for another job; it is re-downloadable (sha256 bd08f219…) and will be re-fetched for the sweep. The sweep is queued behind the Scout training.
• 42-task agentic-coding benchmark: 42/42, 0 voids, 184.2 min, 119,938 output tokens — see Myric/apex-quant-benchmarks. Against the previous Nemotron-3-Super (42/42) that is ~20 min faster and ~28% fewer output tokens at the same score.

Validation

Per the house quality standard, a public quant carries five checks. Status for this tier:

All five checks pass. The reference is a band-serialized BF16 forward over 64×512 wikitext-2 chunks (Mamba kernel path + explicit causal mask), because the 225 GiB BF16 cannot be resident on the 119 GiB GB10. Sanity check: reference PPL 4.284 vs the quant's 4.2777 (ratio −0.15%), so the divergence is measured against the same distribution, not a broken forward. See GATE.md for the full log and commands.

The model

• 88 layers: 40 Mamba2, 40 latent-MoE, 8 full attention. 512 routed experts, top-22, 1 shared expert; latent 1024; hidden 4096; 32 heads / 2 KV heads × 128; 1M trained context; vocab 131072; 1 MTP layer.
• Per-token KV is 8 KiB (8 attention layers × 2 KV heads × 2×128 dims × f16). The Mamba state is constant-size, not token-scaled.
• Why that matters on a 119 GiB GB10: ~2 GiB KV at 256k and ~8 GiB at 1M. The 81 GB model plus a 1M context still fits (~89 GiB total) with ~30 GiB left for agents/OS; at 256k the total is ~83 GiB. A mixed SWA/global 118B model like Laguna-S needs ~48 GiB of KV at 1M and does not fit. The same 8 KiB/token also means multiple 256k slots are cheap (2 GiB/slot), unlike 12 GiB/slot architectures.

Quant

• 81.03 GB (75.46 GiB), 5.37 bpw, 763 tensors.
• STRUCTURAL allocation (role/band priors). No measured sensitivity: the BF16 base is 225 GiB and does not fit resident on the 119 GiB GB10 for a forward-pass sweep.
• Imatrix: laguna_calib_codeweighted.txt — a code-weighted corpus, not generic Wikipedia — 295 chunks × 512, band-serialized (--band 2).
• F32 pins (409 tensors): routers (ffn_gate_inp), exp_probs_b.bias, ssm_a, ssm_d, ssm_norm, ssm_conv1d, ssm_dt.bias, every norm and every bias.
• Type mix: F32 409, Q8_0 160, Q6_K 123, Q5_K 31, IQ4_NL 31, Q5_0 9.
• Recipe: configs/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-i-quality.recipe
• Imatrix: NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.imatrix

Source

• Text BF16: ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF (NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf, 241,518,661,568 bytes, sha256 bd08f21911423a9c0a54794c7abce1c553ac5d972aad1c73bb903d0295e44eaf).
• Same repo: vision mmproj (BF16 1.73 GB) and MTP head (BF16 8.04 GB / Q8_0 4.28 GB / Q4_0 2.27 GB). Both are separate from this APEX quant.

Files

The quant is split into two byte-identical parts to fit Hugging Face's 50 GB/file limit:

• NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-APEX-i-quality-split.gguf-00001-of-00002.gguf
• NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-APEX-i-quality-split.gguf-00002-of-00002.gguf

Keep both in the same directory and load part 00001; llama.cpp finds the rest from the filename. The unsplit 81.03 GB file is the same quant.

Run

llama-server -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-APEX-i-quality-split.gguf-00001-of-00002.gguf \
  -ngl 999 -fa on --jinja \
  --ctx-size 65536 --parallel 1 \
  --temp 1.0 --top-p 0.95 --top-k 20

• Always set --ctx-size explicitly. The model was trained to 1M; leaving it unset asks llama.cpp to size the KV cache for the trained context.

• Vision: add the ggml-org mmproj with --mmproj ...; it is not in this repo.

• MTP (recommended): the native head is a separate file. Load it with --spec-type draft-mtp --model-draft mtp-...gguf --spec-draft-n-max 2. Measured on the 5-task quality slice (temp 1.0, reasoning high, 262k ctx):

  config	pass	wall	output tokens
  no drafter	5/5	28.4 min	23,203
  MTP Q4_0, n_max 2	5/5	16.6 min	19,294

  1.72× wall-time speedup, 5/5 both. Short-prompt decode 18.5 → 32.3 tok/s (1.74×), acceptance 86.6–90.7%, mean accepted length 2.81. Use a Q4_K head for the published recipe (Q4_0 was the quick test; same size class). LiLiCorr/DSpark/EAGLE/ n-gram remain untested or blocked; see TODO.md.

• Build: needs a llama.cpp with nemotron_h_moe support (mainline 2026‑10+).

Attribution

• Base: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 (NVIDIA Open Model Development License / openmdw-1.1 — see the base repo).
• BF16 source: ggml-org/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF.
• Engine: llama.cpp (MIT).
• APEX: localai-org/apex-quant (MIT).

Unofficial community quantization; not affiliated with or endorsed by NVIDIA.