Support on Ko-fi or X Money: @DJLougen

NVIDIA-Nemotron-3.5-Bolt-E4B-MLX

NVIDIA-Nemotron-3.5-Bolt-E4B quantized for MLX — 5 variants, 5.2–15.9 GB (5.19–16.00 bits per weight), measured peak RAM 4.9–14.9 GiB

MLX conversions of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 (itself based on google/gemma-4-E4B-it). Built with mlx-vlm 0.7.10. Each variant ships in its own subfolder (bf16, 8bit, 6bit, dec6-emb4, 4bit).

Downloads

Official model source: nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 — please credit NVIDIA for the original model.

At a glance

Peak memory is the measured MLX peak on an M3 Max (short prompt, and a ~3.7k-token prompt), not a device-fit claim. Judgement labels are computed from the measured top-1 agreement: PERFECT ≥ 95%, GREAT ≥ 80%, GOOD ≥ 50%, MISS = below that or a failed smoke test.

Quick start

pip install mlx-vlm

Subfolder note: mlx-vlm 0.7.10 has no subfolder parameter — mlx_vlm.load() / get_model_path() accept either a local directory or a repo id (+ git revision), so point it at a local clone of one subfolder, or download the subfolder first. Verified end-to-end: hf download DJLougen/NVIDIA-Nemotron-3.5-Bolt-E4B-MLX --include "4bit/*" --local-dir e2e-check then python -m mlx_vlm generate --model e2e-check/4bit --prompt "What is 17*23? Answer with just the number." --max-tokens 8 printed 391.

The quick-start example uses dec6-emb4 — the smallest variant currently rated GREAT on the measured numbers. Swap the subfolder name for any variant in the table above.

hf download DJLougen/NVIDIA-Nemotron-3.5-Bolt-E4B-MLX --include "dec6-emb4/*" --local-dir bolt-e4b-mlx

python -m mlx_vlm generate --model bolt-e4b-mlx/dec6-emb4 \
  --prompt "Explain what a QR code encodes." --max-tokens 256

Or from Python: from huggingface_hub import snapshot_download; snapshot_download("DJLougen/NVIDIA-Nemotron-3.5-Bolt-E4B-MLX", allow_patterns=["dec6-emb4/*"]) returns the snapshot dir; the variant then lives in <snapshot>/dec6-emb4.

Exact numbers

Click to expand the full measurement table (KL vs MLX bf16; 8,192 tokens, 16 seqs × 512 tokens, Apple Mac M3 Max 36GB (darwin 27.2.0, arm64), 2026-10-23)

gen_tok_s is wall-clock generation speed including prefill and warm-up; the decode-only tokens/s reported by mlx-vlm is in the results file (gen_tps_reported). top1_n is the number of positions where the variant's greedy argmax token equals the bf16 argmax token (7,926 of 8,192 for 8bit = the 96.75% shown). Memory basis, verbatim from the results file: active_after_load_gib = mx.get_active_memory() after load; peak_mem_gib = mx.get_peak_memory() after mx.reset_peak_memory() before a 256-token greedy chat-template generation from the short prompt; peak_mem_gib_4k = same after reset, but with a 3683-token templated prompt (+256 generated). Prefill chunked at 2048 tokens.

How it was made

Converted from the upstream BF16 checkpoint with mlx-vlm 0.7.10. The plain variants (8bit, 6bit) are straight affine quantization at group size 64:

python -m mlx_vlm convert --hf-path nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 \
  --mlx-path out/8bit -q --q-bits 8 --q-group-size 64
# 6bit: --q-bits 6

dec6-emb4 uses a custom quantization predicate (full recipe below; the predicate/eval script ships in results/ of this repo):

custom predicate over mlx_vlm convert(): 6-bit group-64 affine for every tensor under language_model.model.layers.* (attention q/k/v/o, MLP gate/up/down, AND the per-layer PLE gate/projection linears — they live under layers.N and match the predicate); 4-bit group-64 for the only 4 tensors outside layers: embed_tokens, embed_tokens_per_layer, per_layer_model_projection, embed_audio.embedding_projection; vision/audio towers unquantized; lm_head tied to embed_tokens. Quant map: 342x6-bit + 4x4-bit. Build script: results/build_dec6_emb4.py

Quality was measured against the MLX BF16 conversion (not the NVIDIA original) as the comparator: KL divergence and top-1 agreement over 8,192 tokens of text (deterministic 60/20/20 interleave of WikiText-2 test prose, CPython 3.14 stdlib argparse.py source, and GSM8K train questions+answers), plus a greedy 17*23 arithmetic smoke test per variant. The MLX bf16 conversion itself was measured against the upstream transformers BF16: KL 0.0027, top-1 agreement 98.1% on a 2048-token subset — so the quant-KL numbers are real quantization loss, not conversion error. The full comparator path, prompt, and packing details are in card_data.json in this repo.

Recipes we measured and didn't ship

Click to expand: measured variants excluded from this release

Limitations

• No task accuracy suite was run — no MMLU, GSM8K, or any benchmark scores.
• Quality is proxied by KL divergence and top-1 agreement vs the MLX BF16 conversion only.
• One machine, one text set, one seed; vision and audio behaviour were not evaluated.

License & credits

Released under the openmdw-1.1 license (inherited from the upstream model). All credit for the original model goes to NVIDIA (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, based on google/gemma-4-E4B-it).

Support on Ko-fi or X Money: @DJLougen