Nemotron 3.5 Super LiLiCorr for MLX — dequantized BF16 storage

This is the LiLiCorr draft sidecar for NVIDIA Nemotron 3.5 Super VL 120B A12B. It requires the matching target model and the engine support in mlx-serve PR #870. It does not run as a standalone chat model; generic mlx-lm or mlx-vlm standalone loading is not provided here.

The weights were dequantized from NVIDIA's released NVFP4 checkpoint into BF16 storage. They retain the source checkpoint's quantization error. This is not NVIDIA's original full-precision training checkpoint, and the conversion does not recover precision lost during upstream quantization.

Source and contents

• Source: NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr, pinned revision 7c7221aeb4cb3139307aa457f7dce11a8fb6ceb2.
• Matching target family: Nemotron 3.5 Super VL 120B A12B. The engine was validated with Duckets' Super MLX 4-bit conversion.
• Five draft transformer layers with grouped dynamic convolutions and sliding attention; block size 8.
• Two-layer LiLiCorr head, hidden width 1024, eight heads, eight candidates per draft slot.
• Independent draft LM head included and required: has_own_lm_head: true, vocabulary size 131072. The target verifies proposals with its own head.
• Seven BF16 safetensors shards, 124 tensors, 3,328,759,316 weight bytes. No tokenizer, target language model, or vision tower is included.

The target supplies its tokenizer, token embeddings, and captured intermediate hidden states. Its token IDs must match the source Super vocabulary; no vocabulary remapping is performed. Image/video inputs are encoded by the target's vision path before its conditioned hidden states reach the sidecar.

Use with mlx-serve

Use a build containing the Super/LiLiCorr changes in PR #870. Replace <target> with the directory containing the matching Super MLX target's config.json:

hf download thelogicalgate/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-LiLiCorr-MLX-BF16 \
  --local-dir "<target>/drafter"

Load or reload the target to discover its drafter/ sidecar. An explicit CLI example is:

MLX_SERVE_DFLASH_QUANT_BITS=8 \
  mlx-serve --model "<target>" --drafter "<target>/drafter" --serve

BF16 describes the files in this repository. mlx-serve may quantize eligible assistant linears at load time; its general default is 8-bit, with a separate accelerated path selecting 4-bit. The explicit setting above selects 8-bit reproducibly. MLX_SERVE_DFLASH_QUANT_BITS=0 selects the dense path for reference checks.

Drafting can be disabled with --no-drafter, a per-model drafter: "off" setting, or per-request enable_drafter: false. The server's [spec-stats] mode=dflash line reports engagement and accepted draft tokens. This repository makes no general throughput or acceptance-rate claim; both depend on the target conversion, runtime precision, and workload.

Conversion and validation

The conversion script decodes native E2M1 nibbles, E4M3 block scales, and the float32 global scale. It checks every chunk of all 26 quantized matrices against the unmodified, pinned NVIDIA Model Optimizer CPU dequantizer before casting to BF16. Every serialized tensor is then read back and hashed.

Only explicit serving-layout adaptations are applied: dotted attn.in_proj keys, flattened slot/rank embedding matrices, and a materialized relative/same-slot attention bias. The own LM head and convolution tensors are preserved. conversion.json lists every output tensor, source keys, operation, shape, and SHA256.

The full-head verification script executes NVIDIA's actual Super-sized LiLiCorr head using synthetic inputs and the trained source weights. The converted layout gives an identical attention bias and zero start/pair factor difference for two fixed seeds. See verification.json. These checks establish conversion/layout parity; they do not establish new model-quality or performance results.

To reproduce from the mlx-serve source checkout containing those scripts, with PyTorch, Transformers, and Safetensors installed:

hf download nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr \
  --revision 7c7221aeb4cb3139307aa457f7dce11a8fb6ceb2 \
  --local-dir ./source-drafter

curl --fail --location \
  https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/6bf36980e1d73a46750c947b7ff4c3cc3fa9c34d/modelopt/torch/quantization/qtensor/nvfp4_tensor.py \
  --output ./nvfp4_tensor.py
curl --fail --location \
  https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/6bf36980e1d73a46750c947b7ff4c3cc3fa9c34d/modelopt/torch/speculative/plugins/modeling_lilicorr.py \
  --output ./modeling_lilicorr.py

python tests/convert_nemotron_lilicorr.py \
  --source ./source-drafter --out ./converted-drafter \
  --reference ./nvfp4_tensor.py
python tests/verify_nemotron_lilicorr.py \
  --source ./source-drafter --converted ./converted-drafter \
  --nvfp4-reference ./nvfp4_tensor.py \
  --head-reference ./modeling_lilicorr.py --out ./verification.json

Both scripts verify the reference source hashes. The converter additionally verifies the pinned source checkpoint hash and refuses existing output directories.

License and attribution

The model materials retain the upstream OpenMDW License Agreement version 1.1; the full agreement is included in LICENSE. See NOTICE for origin and conversion details and SOURCE_README.md for NVIDIA's unmodified original model card. NVIDIA Model Optimizer reference code has its own source-code licenses, separate from the model-material license.