NVIDIA Nemotron 3.5 Super VL 120B: TurboQuant 3-bit (MLX)

Data-free 3-bit quantization of nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 for Apple Silicon, built with TurboQuant-MLX. Nemotron 3.5 Super VL is NVIDIA's 120B-parameter vision-language model: a hybrid of Mamba-2, attention and a latent Mixture-of-Experts (512 experts, 22 active per token, about 12B active parameters), with a RADIO vision encoder for images and documents.

About 20 tok/s on a 64 GB Mac, with two settings.

• 53.9 GB on disk (50.2 GiB), down from 249 GB in BF16.
• ~19–21 tok/s decode on short answers, 15.5 tok/s with an 18K-token prompt. On 0.29.0 that was about 2.5× the ternary build; with turboquant-mlx-full 0.30.0 the ternary build decodes at a similar ~20 tok/s while the Mac is cool (14–15 once it is hot, measured) at default settings.
• It needs a 64 GB Mac with both of these: the GPU memory limit raised with sudo sysctl -w iogpu.wired_limit_mb=60416, and --prefill-step-size 256. Without the smaller prefill step, a ~4,000-word prompt runs out of GPU memory. With both, it leaves only a few GB for macOS, so close other heavy apps.
• If you'd rather not change any settings, or have less than 64 GB, use the ternary build: NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-tq3a-tqTe-g64 (31.5 GB, default settings, similar speed since 0.30.0).
• Data-free: no calibration set. Experts, attention, Mamba projections, shared experts and the latent projections use a 3-bit Lloyd-Max codebook; embeddings, lm_head and the vision tower are 8-bit affine. Randomized Hadamard rotation, per-group RMS scale, decoded inline by fused Metal kernels.

Documentation: manjunathshiva.github.io/turboquant-mlx: the Nemotron 3.5 Super VL page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents, vision models and memory and macOS settings.

How to run

pip install "turboquant-mlx-full[vlm]>=0.29.0"

# once per boot: let the GPU use up to ~59 GiB (undo with =0, or reboot)
sudo sysctl -w iogpu.wired_limit_mb=60416

python -m turboquant_mlx.generate_vlm --model manjunathshiva/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-tq3-g64 \
    --image invoice.png --prompt "What is the invoice number and the total due?" \
    --prefill-step-size 256 --max-tokens 512 --temp 0.6

turboquant-serve-vlm --model manjunathshiva/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-tq3-g64 --prefill-step-size 256 --port 8080

Sampling: NVIDIA recommends temperature 1.0 / top_p 0.95 for text and 0.6 / 0.95 for vision.

Reasoning is off unless you ask for it (enable_thinking=True, or chat_template_kwargs on the server).

Requests to the server must name a model (mlx-vlm answers 422 without one).

Will it fit my Mac?

Measured on a 64 GB M4 Max with iogpu.wired_limit_mb=60416 (a GPU limit of ~63.3 GB):

• 64 GB Mac: yes, with the sysctl bump and --prefill-step-size 256.
• Less than 64 GB: no. Use the ternary build.

Quality

The same battery as the ternary build, at NVIDIA's recommended sampling, reasoning off:

These are spot checks, not benchmark scores; there is no BF16 reference on a Mac to compare against.

Provenance and licence

• Base model: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, by NVIDIA, licensed under the OpenMDW License Agreement 1.1. A copy is in LICENSE.md; NVIDIA's notices are retained in NOTICE.
• This repo contains a quantized derivative (weights re-encoded, nothing retrained). The MTP head is not included. NVIDIA's custom Python code is not included: mlx-vlm provides the model and processor.
• Converted with python -m turboquant_mlx.convert_vlm --bits 3 --group-size 64 --quantize-extras (turboquant-mlx-full 0.29.0).
• For NVIDIA's model details, evaluations, bias, privacy and safety notes, see the base model's card.