Nemotron 3.5 Super VL EXL3 4.22 bpw (with native MTP head)

Summary

This is an EXL3 quantisation of Nemotron 3.5 Super VL (nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) by NVIDIA, architecture NemotronH_Omni_Reasoning_V3.

• Quantised to 4 bits per weight. The language-model linear layers of the 88 trunk layers are EXL3 at 4 bits per weight: the routed and shared experts, and the Mamba, attention and latent projections.
• The MTP head in 4 bits (this revision). The mixture-of-experts block of the multi-token-prediction (MTP) head, 1,028 matrices, is EXL3 at 4 bits per weight in its own shard (MTP head). The previous revision, b546ce51 (REXL3 v0.1.0), carries the bf16 head.
• Kept in the original bf16. Embeddings, output head, routers, norms, Mamba controls, the vision tower and the projector, and the MTP head's attention block, eh_proj, router and norms.
• Size. 61.2 GiB of tensor data and 64.4 GiB on disk: 4.22 bits per weight over all 124.5 B parameters. The 122.4 B quantised parameters take 4.02 (rate 4 plus the fp16 scales), the 2.1 B kept in bf16 take 16.
• Fidelity to the bf16 original, on text (14 samples x 1,024 tokens): KL mean 0.0348 nats, median 0.0045, top-1 agreement 93.0 % [1]. This is a small set; Quantisation fidelity says what it does and does not show.
• Repair overlay. The first encode of 715 expert matrices failed, so those matrices were re-encoded and ship in a separate overlay shard named by the index (Repair overlay). Load this repository through model.safetensors.index.json, not by globbing shards.

REXL3 runtime: coming soon.

Model

Architecture facts from config.json [3]:

The configuration, tokenizer, chat template, processor configuration and custom code are the base model's, unmodified.

What is quantised and what is not

Quantiser. ExLlamaV3 at commit 151539c7, ordinary LDLQ, mul1 codebook, rate 4 for every eligible language and expert projection. The first encode used encoder seed 7; the repair searched the seed per matrix (below).

Calibration. Input factors (Hessians) came from a fixed 256-example multimodal pilot corpus, passed through the original bf16 blocks. One exception: 310 matrix groups of layer 39 were re-encoded after a crash, and their input factors were taken from the embedding stream instead of the stream propagated through blocks 0 to 38. For those groups [4]:

• the weight error is typical (rel_l2 about 0.08);
• the output error on pilot inputs is 4 to 6 times that of the neighbouring groups;
• on text outside the pilot corpus they are no worse than their neighbours.

Precision hold. Six routed expert matrices are stored as the byte-identical bf16 originals instead of being encoded: one expert pair in each of layers 26, 39 and 81. They were kept in bf16 by the quantiser's precision-hold policy. The reason recorded when the pack was made (no routed traffic in the calibration corpus) has not been re-checked for this card.

MTP head

Revision e3857098d4ebb503eeb4ca425ad5e7628ee9663f replaces the bf16 MTP head's mixture-of-experts block (5.3 GiB) with an EXL3 4 bpw encoding (1.34 GiB) in model-head4-00001-of-00001.safetensors (1,443,681,760 bytes, 4,112 tensors); model.safetensors.index.json maps the head's tensors to it. HEAD4_MANIFEST.json records:

• every one of the 1,028 matrices, with its weight error against bf16: rel_l2 median 0.0870, p90 0.0968, max 0.1016 (none re-seeded);
• the calibration: the head's own inputs, 92 sequences and 144,406 token pairs, every expert reached (164 to 31,555 rows a matrix);
• the source revision's index (d2c82b14..., revision b546ce51) and the sha256 of the new shard and index.

The head's attention block, eh_proj, router and norms are the original bf16 tensors. model-00005-of-00032.safetensors and model-00008-of-00032.safetensors held only the bf16 head's tensors and are not part of this revision; every tensor the index names is present. The MTP head drafts tokens that the model then verifies, so its precision changes how many drafts are accepted, not what is generated (with REXL3 the seeded replies are byte-identical with either head).

Repair overlay

The first encode (seed 7) failed on 715 matrices: 710 routed expert projections and 5 shared-expert projections, mostly in layers 3 and 5.

• Cause. The encoder's global scale search collapsed on some random sign-flip draws [4]. Calibration coverage was not the cause.
• Damage. The failed matrices had a weight error (rel_l2 against bf16) of 0.24 to 2.61, against a typical 0.087.
• Fix. Each one was re-encoded at 4 bpw with the same call and the same input factor, with the seed searched per matrix. No matrix was moved to bf16 [1][2].

The layout of the repair:

• model-overlay-00001-of-00001.safetensors (1,037,947,276 bytes) holds the 715 re-encoded matrices (2,860 tensors). model.safetensors.index.json maps those tensor names to the overlay.
• The original shards (30 of the 32 are in this revision) still contain the failed copies under the same names. A loader that reads every model-*.safetensors file instead of following the index can pick up a failed copy.
• 2,684 tensors of layers 32 to 34 appear in two of the original shards with identical bytes. This is harmless.
• PATCH_MANIFEST.json records:
  • every replaced matrix, with its old and new rel_l2, byte size and source;
  • the overlay's sha256;
  • the sha256 of the index before and after the repair;
  • the revision the overlay was built from, by commit id. That earlier revision is not published.
• VALIDATION.json holds the scalars of the checks below.

Weight-space scan of the 715 replaced matrices against bf16 [1]:

No routed matrix in the whole pack is above 0.3, and the unchanged matrices are identical.

Quantisation fidelity (text)

Setup [1]:

• Metric. KL(bf16 ‖ this pack) in nats, over the full 131,072-token vocabulary, with fp32 log-softmax.
• Data. 14 samples x 1,024 tokens (14,336 positions):
  • code: Python, Rust, CUDA, JavaScript;
  • structured text: JSON, markdown tables;
  • math and logic;
  • languages: German and French, Russian and Japanese;
  • a dialogue and English technical prose;
  • a chat-template prompt with tool definitions and a reasoning turn;
  • two README-text samples.
• Reference. The bf16 checkpoint, run block by block.
• The pack's logits come from an EXL3 implementation of this architecture (REXL3), run on a copy of the weights stored in a different shard layout. That copy holds the same 166,757 tensors as revision b546ce51, byte for byte [3]; this revision changes only the MTP head, which is not part of this KL.

Per sample (KL mean, this revision):

Non-English text, the chat-template prompt and technical prose are the hardest. The repaired pack is within the measurement noise of the ceiling (0.0348 against 0.0343, with a noise floor of about ±0.001 nats) [1][4]. The remaining KL is in the matrices that were never flagged.

What this does not show:

• 14 samples is a small set, and the mean has a heavy tail: the worst few positions of a sample move it. It is not an estimate of KL on open-ended generation, on long context or on other languages.
• The MTP head (its MoE block in 4 bits in this revision) and the vision tower (bf16) are not part of this KL. No image or video KL is reported for this revision.

Limitations

• No formal benchmark. No task benchmark (accuracy on a standard suite) is reported for this pack. Text fidelity rests on the 14 x 1,024-token KL above.
• Narrow calibration. The calibration corpus was a narrow 256-example pilot. Matrices calibrated on it have 3 to 7 times higher output error on text outside it [4]. A broader re-calibration was not done. Layer 39's 310 groups used the embedding stream (see What is quantised).
• MTP head. Its MoE block is 4-bit in this revision; its weight error is in HEAD4_MANIFEST.json. No KL or task benchmark of the head on its own is reported here. Revision b546ce51 keeps the bf16 head.
• Load by index. The original shards (30 of the 32 are in this revision) still hold the failed first-encode copies of the 715 replaced matrices. Only the index decides which copy is read.
• Bundled transformers code. The HF transformers code in this repository (modeling_*.py) is the base model's and does not load EXL3 weights. Its torch Mamba-2 fallback (used without mamba_ssm) gave wrong logits after 128 tokens in transformers 5.12.1 [4].
• Untested loaders. Loading this architecture with stock ExLlamaV3 has not been tested.

Files

The repository has 32 referenced safetensors files (30 of the 32 first-encode shards, the overlay and the MTP head shard) and is 64.4 GiB on disk.

The base model's generation_config.json sets temperature 1.0 and top-p 0.95.

SHA-256

CHECKSUMS.sha256 lists every file except itself, .gitattributes and this README.md (the card can be edited after release). Verify a download with sha256sum -c CHECKSUMS.sha256. Key files:

db72d175d6ae44443584a5c560bfd411a3cfbef9984287ed8b83326b9b092fc1  model-overlay-00001-of-00001.safetensors
56f1292b57dd84c2f4e132dac48a661178d3381b03e124bf7c4dc847e676f16b  model-head4-00001-of-00001.safetensors
eb0f1adacd51688a978b1b3175bdf1e723dce211f594c22c21a49a8c6e822641  model.safetensors.index.json

Revisions

Both revisions resolve; a later commit that changes only README.md keeps these weights. Between them only the MTP head changed: model-head4-00001-of-00001.safetensors and HEAD4_MANIFEST.json were added, model-00005-of-00032.safetensors and model-00008-of-00032.safetensors removed, and the index and CHECKSUMS.sha256 updated; the trunk shards, the overlay, the vision tower and the base model's files are the same.

Sources

1. VALIDATION.json, in this repository: the KL against bf16 and the weight scan.
2. PATCH_MANIFEST.json, in this repository: the 715 replaced matrices.
3. This repository's config.json and safetensors headers (computed). The copy of the weights used for [1] was compared with revision b546ce51 tensor by tensor: 166,757 of 166,757 identical in dtype, shape and sha256 of the raw bytes.
4. Quantisation and repair notes for this model (REXL3 engineering notes, 2026-10-10 to 2026-10-12, not yet public).

License

These weights are a derivative of Nemotron 3.5 Super VL (nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) by NVIDIA and are distributed under the base model's license, the OpenMDW License Agreement, version 1.1. As that license requires of redistributors, this repository retains a copy of the license, in LICENSE, and the copyright notice and other notices of origin of the base model that apply to this distribution:

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

The base model's configuration, tokenizer, chat template, processor configuration and custom code files are included unmodified, with whatever notices they carry.

Six of the base model's custom code files carry their own NVIDIA copyright notice and the Apache License, Version 2.0, header: configuration_nemotron_h_omni.py, configuration_radio.py, image_processing_nemotron_h_omni.py, modeling_nemotron_h_omni.py, modeling_radio.py and processing_nemotron_h_omni.py. They are redistributed unmodified with a copy of that license (LICENSE-Apache-2.0.txt) and are listed in NOTICE.

Modifications (a statement of what this repository changed, given for clarity):

• The language-model linear weights were quantised to EXL3 at 4 bits per weight with ExLlamaV3 (commit 151539c7). 715 of them were re-encoded as described under Repair overlay.
• The MoE block of the MTP head (1,028 matrices) was quantised to EXL3 at 4 bits per weight (model-head4-00001-of-00001.safetensors, HEAD4_MANIFEST.json).
• 6 expert matrices, the MTP head's attention block, eh_proj, router and norms, the vision tower, the projector, embeddings, output head, routers and norms are the original bf16 tensors.
• README.md, NOTICE, PATCH_MANIFEST.json, VALIDATION.json, HEAD4_MANIFEST.json and the checksum files are new. No other file was changed.

The EXL3 format and encoder are from ExLlamaV3 (MIT, Copyright (c) 2025 Turboderp). No ExLlamaV3 code is included in this repository. The limitations and disclaimers of the base model's card apply.