NVIDIA-Nemotron-3.5-Bolt-74B-A9B — 4bit
MLX conversion of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16, pinned to revision bca0a8cb74200a2eb7ca56eb8428b821c829bd6e. Converted and tested on an Apple M5 Max with 128 GB unified memory using MLX-VLM.
Affine weight-only quantization, 4 bits, group size 64. The vision encoder remains BF16; eligible projector linear weights are quantized. This model accepts text, images, and video and generates text.
| Property | Value |
|---|---|
| Parameters | 74B total, 9B active |
| Architecture | Mamba-2 / Transformer hybrid MoE with C-RADIO vision encoder |
| Weight files | 43.15 GB (decimal) |
| Source context limit | 262,144 tokens |
| Native MTP head | Not included in the upstream Bolt checkpoint |
Usage
Install the tested MLX-VLM revision (includes the Nemotron 3.5 port):
pip install 'git+https://github.com/Blaizzy/mlx-vlm.git@e17c49a5061b4399df3f5bda206ea2a63d45486a'
python -m mlx_vlm generate \
--model nativ-community/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-4bit \
--prompt "Write a short Python function that returns the first n Fibonacci numbers." \
--thinking-mode disabled \
--max-tokens 512 \
--temperature 1.0 --top-p 0.95
For images, add --image /path/to/image.jpg and use a prompt such as "Describe this image.". For video, use --video /path/to/video.mp4 --fps 2 with a video question. The included processor and chat template handle media tokens. Use --thinking-mode enabled for reasoning, with a larger output budget.
Conversion and validation
python -m mlx_vlm convert \
--hf-path nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 \
--revision bca0a8cb74200a2eb7ca56eb8428b821c829bd6e \
--mlx-path ./NVIDIA-Nemotron-3.5-Bolt-74B-A9B-4bit \
--dtype bfloat16 -q --q-mode affine --q-bits 4 --q-group-size 64
The saved checkpoint was reloaded and tested with MLX 0.32.4. Text generation returned 4 for 2 + 2; image generation identified a red square on the left and a blue circle on the right; video generation identified a red-to-blue transition.
On the Apple M5 Max, a 128-token greedy code sample decoded at 71.7 tokens/s. Peak MLX memory across the short text, image, and video checks was 46.61 GB. Memory and speed depend on prompt length, media resolution, and generation settings; the source context limit was not benchmarked. See conversion_info.json for provenance and measurements.
License and attribution
The original model is by NVIDIA Corporation. These weights retain the OpenMDW-1.1 license. See NOTICE for conversion details and the original model card for training, evaluation, and intended-use information.