中文Sign up

divinetribe/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-MLX-8bit

Source: 🤗image-text-to-textopenmdw-1.174.9B params9B activated150 GBsafetensors✓ 18 checksumsupdated 4d ago
or submit a torrent

Nemotron-3.5 Bolt 74B-A9B (8-bit, MLX, text + vision)

NVIDIA's Nemotron-3.5 Bolt 74B-A9B converted for Apple Silicon with text, images and video all working. Nothing is dropped: all 28,725 tensors in NVIDIA's checkpoint map onto this build.

This is the 8-bit MLX build, with text, images and video all tested end to end. Stock mlx-vlm can't load this model yet. Bolt uses a Puzzle-searched hybrid language model (Mamba2, attention and MoE layers with a different expert count per layer) plus a RADIO vision encoder. I wrote a nemotron_h_omni model for mlx-vlm to run it, and it ships in this repo.

Precision: 8-bit affine, group size 64. The vision tower, the vision projector and the output head stay bf16. Size: about 81 GB on disk, 1,863 tensors. It needs a 128 GB Mac. Speed: 37 to 41 tokens/s on an M5 Max with 128 GB. Thinking: on or off through the chat template (enable_thinking).

Run it

pip install -U mlx-vlm torch torchvision
hf download divinetribe/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-MLX-8bit --local-dir Bolt-8bit
python Bolt-8bit/mlx_vlm_port/install_port.py
python -m mlx_vlm generate --model Bolt-8bit --image your_image.jpg --prompt "Describe this image."

install_port.py copies the nemotron_h_omni model into your installed mlx-vlm and registers its chat format. It's safe to run twice. Once mlx-vlm supports this model natively, you can skip it.

How it was checked

These checks compare against NVIDIA's own reference code from the base repo:

  • Weights: every checkpoint tensor maps onto the MLX model by name with the right shape, in both directions.
  • Processor: chat template text, token ids (300 image tokens for the test photo) and pixel values match exactly, with thinking on and off.
  • Vision: on a real photo, the RADIO tower reaches cosine 0.99998 against an fp32 reference and the projector 0.99995.
  • Language model: comparing all 88 layers one at a time against fp32, cosine stays at 0.9993 or higher. Final logits reach cosine 0.999997 with KL 0.00075, and the top-1 next token matches.
  • Video: after pruning, the MLX video embeddings reach cosine 0.99981 against NVIDIA's fp32 reference, and the pruning mask agrees with NVIDIA's step on 100% of tokens.

What it can do, tested on this 8-bit build

All 16 checks below pass in a fresh venv with mlx-vlm 0.7.8 from PyPI plus install_port.py, on a 128 GB M5 Max.

Capability What it said
Video, 12 s clip of numbered color cards "3 on red, 7 on blue, 1 on green, 9 on yellow, 4 on purple, 6 on orange", exactly right
Video, motion "The square moves from left to right and changes from red to blue."
Video, 90 s / 180 frames Read the last number (90) and found the one frame with a yellow circle (57)
Several images in one prompt "Image 1: 3 blue circles / Image 2: 5 green triangles / Image 3: 2 red squares"
Photo Two tabby cats on a pink couch plus the two TV remotes, counted correctly
Document / OCR All seven fields of a test invoice read exactly
Chart All five bar values read exactly, plus highest, lowest and sum
Tool calling Well-formed <tool_call> with the right arguments, thinking on and off; it uses the tool result in its answer and skips tools when they aren't needed
JSON Valid on the first try, no code fence, every field right
Languages Spanish, Japanese and Chinese questions answered correctly
Multi-turn with an image Still answers from the image two turns later
Long context Found a hidden number at 32K and at 128K tokens
Reasoning Bat-and-ball answered correctly ($0.05), thinking on and off

Speed: 37 to 41 tokens/s generating. Prompt processing runs 550 to 950 tokens/s with images and about 755 tokens/s at 128K. Memory: 81 GB loaded, peaking around 89 GB on 90 seconds of video and 91 GB at 128K context. Only 8 of the 88 layers are attention, so a 128K cache is about 1 GB.

Video uses NVIDIA's token pruning (0.7 by default, set per call with video_pruning_rate=). One prompt takes images or a video, not both, which matches NVIDIA's reference. Clips longer than 180 frames weren't tested. The model has no audio tower.

License

The weights come from NVIDIA under the OpenMDW 1.1 license and are redistributed here under the same terms. See the base model card for training data, intended use, bias and safety notes. This conversion doesn't change the model's behavior or alignment.

Part of Claude Code Local

This model is one of the fighters in Claude Code Local (3.2k★), which runs Claude Code 100% on-device on Apple Silicon through an MLX-native Anthropic-API server. Not sure which local model to run as an agent? Check the Agent-12 local agent leaderboard: real agent tasks, judged by the filesystem, same hardware for every row.

Built by Matt Macosko in Humboldt, CA. Open to work on local-AI and Apple Silicon inference: matt@ineedhemp.com.