Nemotron 3.5 ASR Streaming 0.6B, ONNX export (fp16, community)

NVIDIA's newest multilingual streaming ASR, running anywhere ONNX Runtime runs. No PyTorch, no GPU required.

• 🌍 One 600M model, 40 language-locales, with automatic language detection built in. The model emits an <xx-XX> tag per utterance, so you don't need a separate language-ID component.
• ⚡ True cache-aware streaming: configurable latency (80 ms to 1.12 s chunk sizes), punctuation and capitalization natively in the stream.
• 💻 3.8x faster than real-time on CPU (RTF 0.26 fp32, 6 real meeting recordings, onnxruntime CPU, no CUDA).
• 🎯 WER 0.0137 vs the PyTorch reference (fp16 build, 6 files, streaming mode). This package reproduces the original model, and we measured it.
• 📉 Competes with Whisper large-v3 at about 1/3 of the parameters, and large-v3 can't stream.

Unofficial community export by @codavidgarcia. Weights remain NVIDIA's, under the OpenMDW-1.1 license (see LICENSE-OpenMDW-1.1 and NOTICE). Not affiliated with or endorsed by NVIDIA.

Why this exists

nvidia/nemotron-3.5-asr-streaming-0.6b (June 2026) is one of the best open streaming ASR models right now, but it ships as a NeMo/PyTorch checkpoint. That means a multi-GB dependency chain and no easy path into sherpa-onnx-style lightweight runtimes (sherpa-onnx#3573, sherpa-onnx#3408).

For the smallest possible footprint, see also the int4 ONNX build from onnx-community, which targets onnxruntime-genai at a fixed 560 ms chunk. This repo sits at the other point of the same trade-off: near-lossless fp16 (WER 0.0137 vs the original), plain onnxruntime with no extra runtime dependencies, any chunk size, and the full export + engine + validation tooling so you can regenerate and verify everything yourself.

Package contents

Total download: ~2.5 GB. The 320 ms chunk is the latency/accuracy sweet spot (RTF 0.26 vs 0.65 at 80 ms on CPU). Encoders for 80/160/560/1120 ms can be generated with the export tooling (source repo).

Quickstart

pip install numpy onnxruntime soundfile scipy

from nemotron_onnx_streaming import NemotronOnnxStreaming  # from the tools repo

engine = NemotronOnnxStreaming(".", language="auto", chunk_ms=320, precision="fp16")
engine.accept_waveform(pcm_float32_16k)   # feed any number of samples
print(engine.get_partial())               # live hypothesis
print(engine.get_final())                 # flush + final transcript
print(engine.detected_language)           # e.g. <es-ES> in auto mode

One-file CLI:

python nemotron_onnx_streaming.py . meeting.wav --language auto --precision fp16
# text: Actually, the masterminds behind this ...
# detected language: <en-US>
# RTF: 0.287

The engine (nemotron_onnx_streaming.py) and the export/validation tooling live in the companion code repository: github.com/codavidgarcia/nemotron-3.5-asr-streaming-onnx. The engine is deliberately small (about 600 lines, numpy + onnxruntime only) so you can vendor it into other projects, or use it as the reference for a sherpa-onnx C++ port.

Validation (measured)

WER parity vs the 🤗 Transformers reference, 6 real meeting-audio files (3 to 60 s, language=auto), jiwer, lowercase + punctuation stripped, torch 2.13 CPU + onnxruntime 1.27, opset 17:

Chunk-size sweep (fp32 graphs, parity around 1e-6 vs HF streaming on all sizes):

_{NVIDIA's FLEURS numbers by chunk size (image: NVIDIA). Accuracy improves with chunk size and stays competitive at 80 ms.}

Graph-level parity with the original cache-aware streaming path: max abs diff below 1.5e-05 over first + 5 steady chunks. Text-level parity near-verbatim at 320 ms and 1120 ms (cross-checked against HF streaming at the same lookahead on 60 s of real meeting audio).

A note on what "parity" means here: it is the WER between this ONNX export and the original model on the same audio, so it measures the fidelity of the conversion. It does not measure absolute ASR accuracy. For accuracy benchmarks see NVIDIA's FLEURS tables on the base model card (avg LangID WER 8.84 to 10.38 across 19 locales depending on chunk size).

Known limitations

• int8 degrades measurably (WER 0.19 vs fp32). It is fast (RTF 0.15) but we don't ship it as the default. Static quantization with calibration is on the roadmap.
• fp16 on CPU doesn't accelerate vs fp32. It halves download and RAM. The speed win shows on GPU via CUDAExecutionProvider.
• Word-level timestamps are not exposed by this RNNT export.
• The engine is single-stream. Batch streaming is a tooling roadmap item.

License and attribution

• Weights: NVIDIA, OpenMDW-1.1. Use, modification and redistribution permitted with attribution. This file plus NOTICE plus LICENSE-OpenMDW-1.1 satisfy that requirement.
• Code (export/engine/validation in the companion repo): Apache-2.0.

Base checkpoint revision: f3d333391852ba876df169dcc9ba902d25b6ab0b. Exported with transformers>=5.13.0 (Nemotron3_5AsrForRNNT), torch.onnx.export (dynamo=False, opset 17), custom fp16 weight cast.