NVIDIA Nemotron 3.5 Super VL 120B: TurboQuant ternary (1.58-bit experts, MLX)

Data-free ternary (1.58-bit) expert quantization of nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 for Apple Silicon, built with TurboQuant-MLX. Nemotron 3.5 Super VL is NVIDIA's 120B-parameter vision-language model: a hybrid of Mamba-2, attention and a latent Mixture-of-Experts (512 experts, 22 active per token, about 12B active parameters), with a RADIO vision encoder for images and documents.

The headline: NVIDIA's 120B vision model, resident on a 64 GB Mac at default settings.

• 31.5 GB on disk (29.3 GiB), down from 249 GB in BF16, about 8x smaller.
• Loads in seconds and stays at 29.3 GiB resident on a 64 GB Mac, with no sysctl memory tweak. Measured peak with a 36K-token prompt: 44.7 GB, under a 64 GB Mac's default ~48 GB GPU limit.
• Text and images: passes our 6-probe text battery and 3 exact-answer vision probes (see Quality), and keeps a 7,400-token reasoning trace coherent.
• Data-free: no calibration set. Experts use a ternary {−c, 0, +c} codebook (base-3 trit packing, about 1.6 bits/weight); attention, Mamba projections, shared experts and the latent projections use 3-bit; embeddings, lm_head and the vision tower are 8-bit affine. Randomized Hadamard rotation, per-group RMS scale, nearest Lloyd-Max centroid, decoded inline by fused Metal kernels.

Documentation: manjunathshiva.github.io/turboquant-mlx: the Nemotron 3.5 Super VL page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents, vision models and memory and macOS settings.

How to run

pip install "turboquant-mlx-full[vlm]>=0.30.0"

# one prompt, with an image
python -m turboquant_mlx.generate_vlm --model manjunathshiva/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-tq3a-tqTe-g64 \
    --image invoice.png --prompt "What is the invoice number and the total due?" \
    --max-tokens 512 --temp 0.6

# OpenAI- and Anthropic-compatible server (binds 127.0.0.1 unless you pass --host)
turboquant-serve-vlm --model manjunathshiva/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-tq3a-tqTe-g64 --port 8080

Sampling: NVIDIA recommends temperature 1.0 / top_p 0.95 for text and 0.6 / 0.95 for vision.

Reasoning is off unless you ask for it. The chat template enables thinking only with enable_thinking=True (pass it through chat_template_kwargs on the server). With it on, expect long traces; our math probe thought for 7,442 tokens before a correct answer.

Requests to the server must name a model (mlx-vlm answers 422 without one); use the id GET /v1/models reports.

Will it fit my Mac?

Measured on a 64 GB M4 Max, at the default GPU memory limit (no sysctl), through turboquant-serve-vlm's generation path:

Peak barely grows with context: only 8 of the 88 layers are attention, so the KV cache is about 8 KB per token (0.3 GB at 36K). The ~11 GB above the resident weights is a fixed prefill working set.

• 64 GB Mac: fits at default settings, with headroom.
• 48 GB Mac: the weights fit, but the peak exceeds a 48 GB Mac's default GPU limit, so it needs sudo sysctl iogpu.wired_limit_mb=… raised. Not yet verified on a 48 GB machine.
• 32 GB or less: does not fit.

turboquant-plan --model manjunathshiva/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-tq3a-tqTe-g64 reads the geometry correctly (512 experts, top-22, 8/88 attention layers) but currently projects about 3.5 GB below the measured peak for this architecture; trust the table above.

Quality

Our standard battery, run once at NVIDIA's recommended sampling, reasoning off unless noted:

Known limits: these are spot checks, not benchmark scores. We have not run NVIDIA's benchmark suite on this build, and there is no BF16 reference on a Mac to compare against.

Speed

About 20 tok/s decode while the Mac is cool, settling at 14–15 tok/s once it is hot (after a few long prompts or about ten minutes of continuous work), on a 64 GB M4 Max with turboquant-mlx-full 0.30.0, measured through generate_vlm with greedy prompts (text and one image). That is 2× 0.29.0 on the same prompts in the same state (8.9–10.5 tok/s), with byte-identical output: every token routes through 22 experts in each of 40 MoE layers, and 0.30.0 removed an integer division per weight from the ternary kernels. On 0.29.0 this build ran ~7–8 tok/s on longer prompts, so upgrade:

pip install -U "turboquant-mlx-full[vlm]>=0.30.0"

It now decodes about as fast as the 3-bit build, NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-tq3-g64 (~19–21 tok/s, 53.9 GB), which also needs the sysctl bump and --prefill-step-size 256. This one runs at default settings.

Provenance and licence

• Base model: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, by NVIDIA, licensed under the OpenMDW License Agreement 1.1. A copy is in LICENSE.md; NVIDIA's notices are retained in NOTICE.
• This repo contains a quantized derivative (weights re-encoded, nothing retrained). The MTP head is not included. NVIDIA's custom Python code is not included: mlx-vlm provides the model and processor.
• Converted with python -m turboquant_mlx.convert_vlm --bits 3 --group-size 64 --ternary-experts --quantize-extras (turboquant-mlx-full 0.29.0).
• For NVIDIA's model details, evaluations, bias, privacy and safety notes, see the base model's card.