NVIDIA Nemotron 3.5 Bolt 74B-A9B — MLX IQ2
This is an MLX-layout repack of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.
The source was commit c6cc2e8ffa353403213d6b9b18aaf8eb684d705a.
It is lossless. Every quantized payload is byte-identical to NVIDIA's QAD-trained IQ2 checkpoint, so nothing was requantized:
| Format | Count | Result |
|---|---|---|
| IQ2_XXS tensors | 27,728 | byte-identical |
| Q8_0 tensors | 113 | byte-identical |
| NVFP4 tensors | 80 | byte-identical |
The exact layout is in SPEC.md. The converter, verifier and fixture scripts are in tools/.
Runtime support required. Stock MLX and mlx-lm cannot load this checkpoint yet. A runtime needs:
iq2_xxs: a new quantization mode with kernels. The experts are stored asweightuint32 plus fp16scales, in GGML IQ2_XXS block order.nemotron_h_puzzle: expert counts and widths that vary per layer (block_configs), plus the latent MoE projections.- NVFP4 global scale: applied to the Mamba projections' output.
Contents
| Text model | 22.47 GB. Routed and shared experts are IQ2_XXS (2.06 bits per weight). Mamba in/out projections are NVFP4. Attention, latent projections and lm_head are 8-bit affine (an exact transcode of Q8_0). Embeddings, router and norms are BF16. |
vision/ |
C-RADIO v4-H vision tower plus projector, BF16 and unchanged (1.69 GB). |
| Tokenizer and chat template | Copied unchanged. eos_token_id: [2, 11]. |
drafter/ |
NVIDIA's LiLiCorr speculative-decoding drafter (0.96 GB), repacked the same way. See below. |
fixtures/ |
Golden inputs and outputs for testing IQ2_XXS and NVFP4 kernels. |
Speculative decoding drafter (drafter/)
Bolt has no MTP layers. NVIDIA ships speculative decoding as a separate DFlash/LiLiCorr drafter, included here so that one download gets everything.
| Source | nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr, commit 4c6a5613b6d887194a3a5493e3a18a5b3930bb86 |
| What it is | A 1.1B-parameter block drafter: 5 GQA layers plus a 2-layer LiLiCorr correction head. It proposes blocks of 8 tokens (block_size) that Bolt verifies. It is not a standalone language model. |
| Inputs it needs from the target | Hidden states from target layers [1, 18, 35, 51, 68, 85]. It also uses the target's embeddings and LM head, which are not duplicated here. mask_token_id is 990. See drafter/config.json → dflash_config. |
| Layout | Lossless. NVIDIA's NVFP4 modules (25) use the same MLX nvfp4 + global_scale layout as the target's Mamba projections. Everything else is BF16 under its original name. NVIDIA's card is kept as drafter/README.nvidia.md. |
| Runtime | Needs DFlash/LiLiCorr support, which is separate from the IQ2 work. |
NVIDIA's -24GB-GGUF is a close but not bit-identical export of the same model.
License
The weights are NVIDIA's, released under OpenMDW-1.1. See LICENSE.
For model details, intended use and benchmarks, see the original model card.