Nemotron-3.5-Transcribe-30B-A4B — MLX

nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B for Apple Silicon.

We built a custom Fused SSM Metal kernel to reach 154 tok/s on a relatively small macbook. We also quantized the model's weights to only use ~17 GB of memory, the best WER vs size was found by quantizing to 3-bit experts, 4-bit LLM, and 8-bit encoders, resulting in a ~0.1 points penalty on our datasets.

You're probably better of using a smaller model, this is just a research experiment.

pip install -r requirements.txt
python transcribe.py audio.wav