Nemotron 3.5 ASR Streaming 0.6B — MLX fp16 for Apple Silicon (320 ms chunks)
MLX build of
nvidia/nemotron-3.5-asr-streaming-0.6b
(cache-aware FastConformer-RNNT with a language prompt, 40 locales). This is an unofficial
build. It was not produced or endorsed by NVIDIA. Weights are under NVIDIA OpenMDW-1.1.
The encoder runs on the Apple GPU through MLX,
and the runtime is pure Rust (mlx-rs), no Python. The small RNN-T decoder and joiner
run on the CPU as ONNX (onnxruntime).
This split lets one GPU serve many concurrent calls: each call's streaming caches stay on
the GPU, and chunks from different calls are batched into one encoder step.
Measured on an Apple M2 Max (30-core GPU, 32 GB)
| ONNX fp32, CPU | this MLX fp16, GPU | |
|---|---|---|
| FLEURS uk WER, greedy, uk-UA prompt (50 clips, 940 words) | 15.43% | 15.43% |
| single-stream RTF (encoder + decode) | 0.306 (1 core) | 0.089 |
| encoder step, batch 1 / 8 / 16 / 32 calls | – | 19 / 26 / 34 / 53 ms |
| concurrent live calls, p95 latency within +300 ms of 1 call | 8 | 48 |
| encoder weights | 2.46 GB | 1.23 GB |
WER with the uk-UA language prompt (prompt_ids 19), as a server would pass it for Ukrainian;
with auto language detection (101) every variant scores about 2 points worse.
The last row comes from continuous 8 kHz telephone streams in 20 ms packets.
Each stream is measured from the last word before a pause to the final transcript.
48 calls with NEMO_BATCH_MAX=64 and 20 CPU decoder workers; with 12 workers the CPU
decoders, not the GPU, were the limit (32 calls). 64 calls fail.
Numerical parity of the MLX port against the ONNX export: max |Δ| ~1e-5 (fp32) on
encoder_out over 6 streaming steps with two calls batched.
Files
| File | What |
|---|---|
encoder.safetensors |
encoder + prompt_projector + encoder_projector, fp16, original checkpoint names and layouts (642 tensors) |
decoder.onnx, joiner.onnx |
RNN-T prediction network and joiner (fp32, CPU) |
tokens.txt |
SentencePiece pieces by id (13088; blank = 13087) |
nemotron_onnx_config.json |
features, language prompt table, cache shapes |
melfilters.f32 |
128-bin mel filterbank (16 kHz, n_fft 512), raw little-endian f32 |
Because the tensor names and layouts are unchanged, any MLX implementation of the model can load
encoder.safetensors. The loader does the layout changes for MLX itself:
- linear weights are transposed to (in, out);
- conv2d weights become OHWI;
- the depthwise conv becomes (9, C);
- pointwise conv1d weights become linear.
Streaming layout: 56 frames of left attention context, 3 frames of lookahead (320 ms = 4 encoder frames per chunk). The first chunk has 25 mel frames; later chunks have 32 (7 frames of overlap for the subsampling convs).
Run (SVB.ASR.Nemo server, macOS)
cargo build --release -p svb-asr-nemo # on the Mac (Xcode + Metal toolchain)
export NEMO_MODEL_DIR=$PWD/nemotron-3.5-asr-streaming-0.6b-mlx-fp16
export NEMO_MEL_FILTERS=$NEMO_MODEL_DIR/melfilters.f32
export NEMO_ENCODER=mlx
export NEMO_MLX_WEIGHTS=$NEMO_MODEL_DIR/encoder.safetensors
export NEMO_MLX_PRECISION=f16 # f16 | bf16 (f32 gains nothing from an fp16 file)
export NEMO_BATCH_MAX=64 # calls per encoder step
export NEMO_WORKERS=20 # CPU decoder workers (above the core count)
./target/release/svb-asr-nemo
Reproduce
cargo run --release -p nemo-mlx --example export_weights -- \
<hf cache>/nvidia/nemotron-3.5-asr-streaming-0.6b/model.safetensors \
out/encoder.safetensors f16
The decoder and joiner come from the fp32 ONNX export (scripts/export/export_onnx.py).
Related builds: ONNX int8 (CPU), ONNX fp16 (GPU).