Nemotron 3.5 ASR Streaming 0.6B — LiteRT

A LiteRT conversion of nvidia/nemotron-3.5-asr-streaming-0.6b (revision 1c8deaecc64b91f034d73e08dd8b64625eb3395d, OpenMDW-1.1), by spybyscript. It is not an official NVIDIA or Google release, and nothing was retrained. For the model itself (training, languages, benchmarks, intended use), see NVIDIA's card, copied as MODEL_CARD.md in each folder.

These are component graphs for a stateful streaming integration, not a drop-in transcriber. The tested profile is en-US, prompt 0, 560 ms chunks and greedy RNN-T. The application owns chunking, caches and decoding, as INTEGRATION.md describes; reference_inference.py is a complete CPU reference.

Versions

Bundle contract. Every folder uses the nemotron35-hf560-v2 contract. It restores terminal punctuation with final-chunk validity masks (args_4 / args_5); see PUNCTUATION_FIX.md. Hosts written for v1 pin revision 153a7dbd809a255e5999130397bcd60a66f81a54.

Manifest SHA-256s:

• fp32 9be792bf2a6ee961020b86f29cfe156591dd884a9aedb96d239282f861692ca9
• fp16 0ea7b854d815682a48ec7eb3632b3a4ec8d2af214f0d51d86e5b986bdb98def1
• int8 996b26e5069bf0b921c2ade14ce10c7bdccee075300a184794554706e27075c8

INT8 changes some outputs and keeps evaluation_only: true. Accept it explicitly by its manifest hash; do not edit the manifest.

Use

pip install numpy soundfile ai-edge-litert==2.2.0 huggingface_hub
python reference_inference.py --bundle fp16 --audio mono-16k.wav
python reference_inference.py --bundle fp16 --encoder-groups npu --audio mono-16k.wav    # the merged encoder graphs
python reference_inference.py --bundle int8 --audio mono-16k.wav --allow-evaluation-bundle

Download one folder plus the root files (allow_patterns=["fp16/*", "npu/encoder*", "*.md", "*.json", "*.py", "LICENSE", "SHA256SUMS"] with snapshot_download). Verify every file's size and SHA-256 against its manifest before loading.

On Android, run each component with LiteRT's CompiledModel and keep every state alive for the session.

• On a Snapdragon NPU, load npu/qualcomm/<SoC>/encoder_*.tflite with the NPU accelerator. These three graphs take the place of the 24 block graphs and the projection.
• Keep the frontend, subsampling, predictor and joint on the CPU. Together they take 13–18 ms a chunk on these phones.
• The merged graphs' inputs and outputs are named in npu/encoder-groups.json and in INTEGRATION.md.

Accuracy

The 20 LibriSpeech test-clean clips below (485 words) are a small clean-speech check, not a benchmark.

On the NPU, 15 of the 20 transcripts are identical to the CPU's. The other five differ by:

• near-homophones ("maid" → "made", "saved our" → "safe are");
• the spelling of rare names;
• one comma.

Against the original model, over 80 clips from 40 speakers (validation-summary.json), the transcripts match on:

• fp32: 80 of 80;
• fp16: 79;
• int8: 68.

Phones

LiteRT 2.3.2 with qnn-runtime 2.50.0, the 20 clips above (171.6 s), at thermal status 0. The frontend, subsampling, predictor and joint run on the CPU in every row. RTF is compute time over audio time; a chunk is 560 ms of audio.

• At FP16 on the GPU, these graphs overflow (wrong or non-finite output). On the GPU, run FP32.
• The NPU needs the merged graphs. With the 24 separate block graphs, each call's fixed cost exceeds its work (7 frames), and loading all 24 slows every call.
• One 24-block graph (1.19 GB) compiles, but the S23 ran out of memory with it.
• The NPU files load in about 1–2 s. The GPU takes 11–16 s to load.

Qualcomm NPU files (npu/)

Runtime:

• LiteRT 2.3.2 with litert-npu-runtime-qualcomm 2.3.2 and Qualcomm's qnn-runtime 2.50.0 (Qualcomm's AI Stack License);
• files precompiled on x86-64 Linux with QAIRT 2.50.0;
• each graph in one NPU partition;
• a file runs only on its chip (Build.SOC_MODEL) and with these runtime versions.

npu/manifest.json lists every file's chip, size and SHA-256.

npu/encoder_*.tflite are the graphs these were compiled from. They use the fp16 blocks' own operators and weights, with the past-cache mask computed in int32 (the NPU has no int64). On the CPU they give the fp16 bundle's tokens on all 20 clips. Other Snapdragons can compile them on the phone with LiteRT's NPU accelerator (not measured here).

Limitations

• Only en-US, prompt 0 and the 560 ms profile were tested. The upstream checkpoint is multilingual, but other languages and chunk sizes have not been validated.
• Errors include names, word substitutions, omissions and punctuation.
• The clean-speech checks above do not establish accuracy in noise, with accents or with specialist terms.

Files

• LICENSE (OpenMDW-1.1) and NOTICE.md
• INTEGRATION.md and PUNCTUATION_FIX.md
• publication.json and validation-summary.json
• SHA256SUMS
• reference_inference.py
• each folder's nemotron-manifest.json, vocabulary, provenance and validation files