Nemotron 3.5 ASR Streaming 0.6B — ONNX int8 for CPU (320 ms chunks)
int8 (CPU) export of
nvidia/nemotron-3.5-asr-streaming-0.6b
(cache-aware FastConformer-RNNT with a language prompt, 40 locales). Unofficial;
not produced or endorsed by NVIDIA. Weights: NVIDIA OpenMDW-1.1.
For x86 CPUs with VNNI/AMX (Intel Xeon Ice Lake and later, AMD Zen 4 and later): dynamic int8 of the encoder's feed-forward MatMuls (per-channel weights, activations quantized per call). Attention, convolutions, the input/output projections, decoder and joiner stay fp32.
| fp32 | this int8 | |
|---|---|---|
| FLEURS uk WER, greedy, uk-UA prompt (50 clips, 940 words) | 15.43% | 16.28% |
| same, 300 clips | 16.02% | 16.27% |
| RTF, 1 core, Xeon 2.1 GHz with AMX | 0.84 | 0.47 (1.78x faster) |
| RTF, 1 core, Apple M2 Max | 0.29 | ~0.27 (use fp32 on Macs) |
| encoder size | 2.46 GB | 1.25 GB |
WER with the uk-UA language prompt (prompt_ids 19), as a server would pass it for Ukrainian;
with auto language detection (101) every variant scores about 2 points worse.
On x86 without VNNI, re-quantize with --reduce-range if accuracy drops.
Graphs
| File | Inputs | Outputs |
|---|---|---|
encoder_320ms_first.onnx |
input_features [B,25,128], prompt_ids [B] int64, cache_mask [B,1,1,60], caches k_cache_i/v_cache_i [B,8,56,128] (i=0..23), conv2d_cache_0..2, conv1d_cache_i |
encoder_out [B,4,640] + updated caches (*_out_*) |
encoder_320ms.onnx |
same, input_features [B,32,128] |
same |
encoder_320ms.data |
weights of both encoder graphs (external data, stored once) | |
decoder.onnx |
token [1,1], h_in/c_in [2,1,640] |
decoder_out [1,640], h_out, c_out |
joiner.onnx |
encoder_frame [1,640], decoder_out [1,640] |
logits [1,13088] |
B is the batch axis (fixed 1). Features: 128-bin log-mel, 16 kHz, 10 ms
hop (nemotron_onnx_config.json has every parameter, the language prompt
table and the cache shapes); tokens.txt maps ids to SentencePiece pieces.
The two encoder graphs are the same network with different input lengths. They share one
weights file, so it is stored and downloaded once. Keep the three encoder_320ms* files together.
The first chunk runs encoder_320ms_first, every later chunk encoder_320ms,
carrying the caches; cache_mask is -1e9 on not-yet-filled cache slots.
Decoding is greedy (or beam) RNN-T over decoder + joiner; blank = 13087.
A complete streaming server that loads this directory as-is
(NEMO_MODEL_DIR): SVB.ASR.Nemo (Rust, onnxruntime).
Reproduce
pip install -r scripts/export/requirements.txt # SVB.ASR.Nemo repo
python scripts/export/export_onnx.py --output-dir fp32 --chunk-ms 320 --validate
python scripts/export/quantize_encoder.py --src fp32 --dst int8 --only-ffn
python scripts/export/share_encoder_weights.py --src int8 --dst int8-flat # one weights file for both encoders
Export code: SVB.ASR.Nemo scripts/export/ (based on
codavidgarcia/nemotron-3.5-asr-streaming-onnx, Apache-2.0).