Nemotron 3.5 ASR Streaming 0.6B — ONNX fp16 for GPU (320 ms chunks)
fp16 (GPU) export of
nvidia/nemotron-3.5-asr-streaming-0.6b
(cache-aware FastConformer-RNNT with a language prompt, 40 locales). Unofficial;
not produced or endorsed by NVIDIA. Weights: NVIDIA OpenMDW-1.1.
For NVIDIA GPUs (onnxruntime CUDA EP). Weights and graph body fp16; all graph inputs/outputs stay fp32, so a runtime written for the fp32 export works unchanged. The encoders have a dynamic batch axis: several streams (each with its own caches, mask and language) run in one call — a batch of 3 equals the one-by-one runs within ~1e-4 (fp32 check at export).
| fp32 | this fp16 | |
|---|---|---|
| FLEURS uk WER, greedy, uk-UA prompt (50 clips, 940 words) | 15.43% | 15.43% |
| encoder size | 2.46 GB | 1.23 GB |
WER with the uk-UA language prompt (prompt_ids 19), as a server would pass it for Ukrainian;
with auto language detection (101) every variant scores about 2 points worse.
GPU throughput numbers will be added once measured.
Graphs
| File | Inputs | Outputs |
|---|---|---|
encoder_320ms_first.onnx |
input_features [B,25,128], prompt_ids [B] int64, cache_mask [B,1,1,60], caches k_cache_i/v_cache_i [B,8,56,128] (i=0..23), conv2d_cache_0..2, conv1d_cache_i |
encoder_out [B,4,640] + updated caches (*_out_*) |
encoder_320ms.onnx |
same, input_features [B,32,128] |
same |
encoder_320ms.data |
weights of both encoder graphs (external data, stored once) | |
decoder.onnx |
token [1,1], h_in/c_in [2,1,640] |
decoder_out [1,640], h_out, c_out |
joiner.onnx |
encoder_frame [1,640], decoder_out [1,640] |
logits [1,13088] |
B is the batch axis (dynamic batch). Features: 128-bin log-mel, 16 kHz, 10 ms
hop (nemotron_onnx_config.json has every parameter, the language prompt
table and the cache shapes); tokens.txt maps ids to SentencePiece pieces.
The two encoder graphs are the same network with different input lengths. They share one
weights file, so it is stored and downloaded once. Keep the three encoder_320ms* files together.
The first chunk runs encoder_320ms_first, every later chunk encoder_320ms,
carrying the caches; cache_mask is -1e9 on not-yet-filled cache slots.
Decoding is greedy (or beam) RNN-T over decoder + joiner; blank = 13087.
A complete streaming server that loads this directory as-is
(NEMO_MODEL_DIR): SVB.ASR.Nemo (Rust, onnxruntime).
Reproduce
pip install -r scripts/export/requirements.txt # SVB.ASR.Nemo repo
python scripts/export/export_onnx.py --output-dir fp32-dyn --chunk-ms 320 --batch dynamic --validate
python scripts/export/convert_fp16.py --src fp32-dyn --dst fp16
python scripts/export/share_encoder_weights.py --src fp16 --dst fp16-flat # one weights file for both encoders
Export code: SVB.ASR.Nemo scripts/export/ (based on
codavidgarcia/nemotron-3.5-asr-streaming-onnx, Apache-2.0).