NVIDIA-Nemotron-3.5-Bolt-E4B-MLX-Serve-mixed-4-8bit

An Apple Silicon MLX conversion of NVIDIA Nemotron 3.5 Bolt E4B, based on the 7.996B-parameter BF16 Gemma 4 E4B checkpoint. It preserves the Gemma 4 vision and audio towers, processor configuration, tokenizer, generation settings, and chat template. The unused frozen shared-KV shard is omitted.

mlx-serve v26.10.x serves text + image; audio input is not yet supported by mlx-serve for E4B.

Pack size on disk: 6.8 GiB. Peak server memory: not yet measured.

Quantization recipe

Uses affine quantization with group size 64: eligible text matrices in dense MLP blocks are 4-bit; attention, token and PLE embeddings, the language-model output embedding, and other eligible text matrices are 8-bit. Vision and audio towers are retained in BF16.

Conversion used mlx-vlm with affine quantization. The source checkpoint is nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16.

Fidelity

Teacher-forced top-1 agreement: 84.8% over 256 positions against BF16 (217 of 256; BOS-prefixed text, candidate dequantized for a CPU forward).

Benchmark

Standard v2 benchmark: not yet run. Smoke test decode on M3 Ultra: about 57 tokens/s (short prompt).

Benchmark tag: nemotron-bolt-e4b-mixed-standard-v2-20261024. This is the local mlx-serve Standard v2 suite on port 11251.

Run with mlx-serve

mlx-serve pull pgmharikrishnan/NVIDIA-Nemotron-3.5-Bolt-E4B-MLX-Serve-mixed-4-8bit
mlx-serve --model pgmharikrishnan/NVIDIA-Nemotron-3.5-Bolt-E4B-MLX-Serve-mixed-4-8bit --serve --port 11234

Send an OpenAI-compatible chat request:

curl http://127.0.0.1:11234/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"pgmharikrishnan/NVIDIA-Nemotron-3.5-Bolt-E4B-MLX-Serve-mixed-4-8bit","messages":[{"role":"user","content":"Write a Python hello world."}]}'

Credit and license

Credit to NVIDIA for Nemotron 3.5 Bolt E4B and its source checkpoint. Use is governed by the OpenMDW License Agreement 1.1; the model is based on Google Gemma 4, which is also covered by the Apache 2.0 license. Review the source model card and license terms before use.