Nemotron-Labs-Unified-ASR-0.6b: Unified ASR model for offline and streaming multilingual inference

Nemotron-Labs-Unified-ASR-0.6b is the multilingual version of Parakeet-unified-en-0.6b — a unified automatic speech recognition (ASR) model that combines offline and streaming inference (with a minimum algorithmic latency of 160ms) in a single model [1]. It extends the English-only unified model to 25 European languages, and is trained primarily on the Granary dataset [5], [6] using the same training data composition as parakeet-tdt-0.6b-v3. The model transcribes speech with punctuation and capitalization support, and uses language-ID prompt conditioning on the encoder to select the target language at inference time.

In our evaluation setup, the new unified model outperforms parakeet-tdt-0.6b-v3 in offline mode and nemotron-3.5-asr-streaming-0.6b in most streaming scenarios with algorithmic latency down to 160ms (see Comparison with already released models).

The new unified model also serves as a strong ASR front end for cascaded streaming speech translation (see Streaming Speech Translation).

Key capabilities:

• One multilingual checkpoint for both tasks: a single unified model covers both offline and streaming inference with a minimum latency of 160ms across 25 European languages.
• Language-ID prompting: the target language is passed to the encoder as a prompt, which stabilizes language selection in low-latency streaming. An automatic language mode (target_lang=auto) is also supported.
• Streaming chunk size flexibility: the streaming latency (chunk + right context) is configurable from 2080ms down to 160ms in steps of 80ms, without re-training.
• Punctuation & Capitalization: built-in support for punctuation and capitalization in the output text.
• Speech translation front end: the right context lets the current chunk use upcoming audio before the transcript is committed, which gives the LLM of a cascaded speech translation system a more accurate source transcript.

Supported Languages:
Bulgarian (bg), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), English (en), Estonian (et), Finnish (fi), French (fr), German (de), Greek (el), Hungarian (hu), Italian (it), Latvian (lv), Lithuanian (lt), Maltese (mt), Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk), Slovenian (sl), Spanish (es), Swedish (sv), Russian (ru), Ukrainian (uk)

License/Terms of Use:

GOVERNING TERMS: Use of this model is governed by the OpenMDW License Agreement, Version 1.1.

This model is ready for commercial/non-commercial use.

Deployment Geography:

Global

Use Case:

This model can be used to transcribe multilingual speech (25 European languages) into text using offline or streaming inference with latency as low as 160ms. It supports a variety of applications, such as voice assistants, live captioning, conversational bots, subtitle generation, and, combined with an LLM translator, streaming speech translation.

Release Date:

Hugging Face 10/27/26

Model Architecture:

Architecture Type: Unified-FastConformer-RNNT with language-ID prompt [1], [2] — an optimized version of the Conformer model [4] with 8x depthwise-separable convolutional downsampling and RNNT loss.

Network Architecture:

• Encoder: Unified FastConformer with 24 layers
• Decoder: RNNT (Recurrent Neural Network Transducer)
• Language conditioning: language-ID prompt fused into the encoder output
• Parameters: 600M

Model Initialization

The encoder was initialized from the parakeet-tdt-0.6b-v3 model to give the unified model strong multilingual acoustic representations from the beginning of training.

Unified offline/streaming training

The model was trained jointly in offline and streaming modes. In the offline mode we use standard offline training with full-context self-attention and non-causal convolutions. In the streaming mode we apply chunked self-attention masks (left, middle/chunk and right context) together with Dynamic Chunked Convolutions inside each FastConformer layer [3] to adapt the model to both decoding scenarios. We also use a mode-consistency regularization (MCR) loss [1] to further reduce the gap between offline and streaming performance. All model parameters are shared between offline and streaming modes (encoder, predictor, and joint networks), including the initial x8 subsampling with non-causal convolutions.

Language-ID prompting

The model uses language-ID prompt conditioning on the encoder, following the approach used in nemotron-3.5-asr-streaming-0.6b. The language prompt is fused with the acoustic representation as follows:

• The FastConformer encoder processes audio into an acoustic embedding of shape (D, T).
• The language encoding expands a one-hot language vector across the time axis → (K, T), broadcasting the language identity to every frame.
• Concatenation along the feature axis produces a fused tensor (D + K, T).
• A projection layer maps the fused features to the RNNT decoder.

Why the prompt is needed. The model uses a single shared SentencePiece tokenizer built from the combined transcripts of all 25 languages, so the tokenizer itself carries no language information and the output token space is shared across languages. In low-latency streaming configurations (latency ≈ chunk + right context ≤ 560ms), the first chunks do not contain enough speech for the model to reliably infer the spoken language, so it can begin emitting tokens belonging to a different language. The model typically self-corrects once enough audio has been accumulated, but the initial tokens are already emitted and cannot be revised. Passing the language ID as a prompt removes this ambiguity from the first chunk onward and makes the transcription language deterministic. In offline mode and at higher streaming latencies the effect is much smaller, since the model has sufficient acoustic context to identify the language on its own.

Automatic language mode. When the spoken language is not known in advance, pass target_lang=auto (the language-agnostic prompt) for a language-agnostic inference. For low-latency streaming we recommend passing the language explicitly, for the reason above. All results in this card are obtained with the explicit language prompt.

Setting up Streaming Configuration

Algorithmic latency is defined as the sum of the chunk size (middle part) and the right context. For the left context we use 5.6s by default (5.6s was used during model training), but other values can be explored for a better accuracy/speed trade-off. Recommended context parameters for different latencies:

The model is quite flexible in terms of context sizes: the values above are recommendations, and other combinations of left context, chunk and right context can be chosen depending on the application. We recommend not decreasing the left context below 2s, as accuracy starts to degrade noticeably beyond that point.

Streaming Inference Limitation

The current streaming inference pipeline is buffered, not cache-aware: the left context is recomputed at every chunk step. Standard NVIDIA streaming ASR models use a cache-aware architecture with causal convolutions and no right context, which allows encoder states to be cached and reused across chunks. The unified model instead relies on non-causal convolutions and a right context to reach its offline accuracy, so the left context has to be recomputed at each step in order to keep training and inference conditions matched. Cache-aware decoding is technically possible with this model, but it breaks that match and leads to accuracy degradation. Fine-tuning the model with zero right context would enable cache-aware decoding, at the cost of the state-of-the-art offline performance that is the main advantage of the unified model. As a result, buffered streaming is computationally more expensive than cache-aware streaming at the same latency. To accelerate the streaming inference, one can reduce the left context to about 3-2s with possible accuracy degradation.

NVIDIA NeMo Framework

How to Use this Model

For now, we provide only inference support for the unified model.

Loading the Model

import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/nemotron-labs-unified-asr-0.6b")

Offline Inference

Pass the language of the audio with target_lang (e.g. "de" for German, "bg" for Bulgarian), or target_lang="auto" to let the model identify it:

output = asr_model.transcribe([wav_file_path], target_lang="de")
print(output[0].text)

Streaming Inference

For streaming inference you can use the stateful chunked RNN-T decoding script from NeMo - NeMo/examples/asr/asr_chunked_inference/rnnt/speech_to_text_streaming_infer_rnnt.py. The left context, chunk and right context are given in seconds (5.6s / 0.56s / 0.56s for 1.12s latency); att_context_size_as_chunk=true enables the chunked self-attention masks; target_lang is the language ID (e.g. en, bg, or auto):

cd NeMo
python examples/asr/asr_chunked_inference/rnnt/speech_to_text_streaming_infer_rnnt.py \
    model_path=<model_path> \
    dataset_manifest=<dataset_manifest> \
    output_filename=<output_json_file> \
    left_context_secs=5.6 \
    chunk_secs=0.56 \
    right_context_secs=0.56 \
    att_context_size_as_chunk=true \
    target_lang=<language_id> \
    batch_size=<batch_size>

You can also run streaming inference through the pipeline method, which uses NeMo/examples/asr/conf/asr_streaming_inference/buffered_rnnt.yaml configuration file to build end‑to‑end workflows with punctuation and capitalization (PnC), inverse text normalization (ITN), and translation support.

from nemo.collections.asr.inference.factory.pipeline_builder import PipelineBuilder
from omegaconf import OmegaConf
# Path to the buffered rnnt config file downloaded from above link
cfg_path = 'buffered_rnnt.yaml'
cfg = OmegaConf.load(cfg_path)
cfg.asr.model_name = "nvidia/nemotron-labs-unified-asr-0.6b"
cfg.lang = "de"  # language ID of the audio
# Pass the paths of all the audio files for inferencing
audios = ['/path/to/your/audio.wav']
# Create the pipeline object and run inference
pipeline = PipelineBuilder.build_pipeline(cfg)
output = pipeline.run(audios)
# Print the output
for entry in output.values():
  print(entry['text'])

---

Input:

Input Type(s): Audio, Language ID
Input Format: Audio waveform (wav), string (language ID)
Input Parameters: One-Dimensional (1D) for audio, One-Dimensional (1D) for language ID
Other Properties Related to Input: Single-channel audio at 16kHz sampling rate, at least 80ms duration is required. The language ID selects the target transcription language from the 25 supported languages.

Output:

Output Type(s): Text String in the prompted language
Output Format: String
Output Parameters: One-Dimensional (1D)
Other Properties Related to Output: The output text includes punctuation and capitalization. The output text might be empty if the input audio doesn't contain any speech.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• NeMo Speech 3.1.0
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
  
• NVIDIA Blackwell
  
• NVIDIA Hopper
  
• NVIDIA Volta
  

Preferred/Supported Operating System(s):

• Linux
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

• Nemotron-Labs-Unified-ASR-0.6b-v1
  

Training, Testing, and Evaluation Datasets:

Training Dataset:

The model was trained on the combination of the Granary dataset's ASR subset [5], [6] and the in-house NeMo ASR Set 3.0, covering 25 European languages:

• 10,000 hours from human-transcribed NeMo ASR Set 3.0, including:

  • LibriSpeech (960 hours)
  • Fisher Corpus
  • National Speech Corpus Part 1
  • VCTK
  • Mozilla Common Voice (v7.0)
  • AMI

• 660,000 hours of pseudo-labeled data from Granary [5], [6], including:

  • YTC (YouTube-Commons)
  • MOSEL
  • YODAS

All transcriptions preserve punctuation and capitalization. Dataset corpora and languages were balanced using temperature sampling. A unified SentencePiece tokenizer, constructed from the training transcripts of all 25 supported languages, is shared across languages.

• Data Modality:

  • Audio and text

• Audio Training Data Size:

  • [10,000 to 1 Million Hours]

• Text Training Data Size:

  • Less than a Billion Tokens

• Data Collection Method:

  • [Hybrid: Automated, Manually-Collected]

• Labeling Method:

  • [Hybrid: Manually-Labeled, Synthetic] - Some transcripts are generated by ASR models, while some are manually labeled

Properties:

Number of data items: about 670k hours of audio data
Modality: Audio
Nature of the content: Personal data and machine generated data
Language: 25 European languages (bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, ru, sk, sl, sv, uk)

Testing Dataset:

• FLEURS (25 languages)

• Mozilla Common Voice (MCV, 13 languages)

• Multilingual LibriSpeech (MLS, 6 languages)

• HuggingFace Open ASR Leaderboard (English), cleaned versions of the test sets:

  • AMI (cleaned)
  • Earnings22 (cleaned, AA-chunked, scored per call)
  • GigaSpeech (cleaned)
  • LibriSpeech test-clean
  • LibriSpeech test-other
  • SPGISpeech
  • VoxPopuli (cleaned)

• Data Collection Method
  

  • [Manually-Collected] - All audios are human recorded
    

• Labeling Method
  

  • [Hybrid: Manually-Labeled, Synthetic] - Some transcripts are generated by ASR models, while some are manually labeled
    

Properties:

Modality: Audio
Nature of the content: Personal data and machine generated data
Language: 25 European languages

Evaluation Dataset:

• FLEURS (25 languages)

• Mozilla Common Voice (MCV, 13 languages)

• Multilingual LibriSpeech (MLS, 6 languages)

• HuggingFace Open ASR Leaderboard (English), cleaned versions of the test sets:

  • AMI (cleaned)
  • Earnings22 (cleaned, AA-chunked, scored per call)
  • GigaSpeech (cleaned)
  • LibriSpeech test-clean
  • LibriSpeech test-other
  • SPGISpeech
  • VoxPopuli (cleaned)

• Data Collection Method:

  • [Manually-Collected] - All audios are human recorded

• Labeling Method:

  • [Manually-Labeled]

ASR Performance

ASR performance is measured using word error rate (WER, %, lower is better). Both reference and predicted text are processed with a text normalizer before scoring. The unified model uses a 5.6s left context in streaming mode, and all results use an explicit language-ID prompt. Latency is the sum of chunk and right-context sizes. The cache-aware Nemotron 3.5 baseline uses its equivalent algorithmic latency. The figure at the top of this card plots these results as WER versus latency.

Comparison with already released models

The multilingual HF subset comprises German, French, Italian, Spanish and Portuguese. The 25-language result uses all available FLEURS, MCV and MLS evaluations (44 dataset-language runs).

Multilingual Speech Recognition

Per-language WER is averaged over the available FLEURS, MCV and MLS datasets for that language. The average row is the mean over all 44 dataset-language runs, the same as in the figure and the comparison table above; because dataset availability differs by language, it is not the plain mean of the per-language rows.

Languages are ordered by accuracy (lowest offline WER first).

English Speech Recognition (Hugging Face Open ASR Leaderboard)

These are the cleaned evaluation sets used for this release: cleaned AMI, GigaSpeech and VoxPopuli; AA-chunked Earnings-22 scored per call; and the standard LibriSpeech and SPGISpeech sets. Therefore, this average is not directly comparable with results computed on the older uncleaned suite. The 0.08s configuration uses a 0.08s chunk and no right context. Without right context the model is less stable, so this configuration was evaluated only for English and is not among the recommended settings.

Streaming Speech Translation (ASR + LLM Cascade)

The unified model can be used as the ASR front end of a cascaded simultaneous speech translation system. The English-only predecessor Parakeet-unified-en-0.6b of this model was the ASR front end for the English-to-X tracks of NVIDIA's IWSLT 2026 system [8], which ranked first by COMET for English-to-German, English-to-Chinese and English-to-Italian translation in both latency regimes. This multilingual checkpoint extends the same approach to speech in any of its 25 ASR languages, provided that the downstream translation LLM supports the requested language pair.

The following results use Riva-Translate-4B-Instruct-v2 with the longest-common-prefix translation policy (waitk=-1) and 800ms end-of-utterance detection on FLEURS. COMET (scored with Unbabel/XCOMET-XL; higher is better) and length-adaptive average lagging (LAAL, seconds; lower is better) are averaged over 23 language pairs in each direction. Maltese is excluded because the translation LLM does not support it.

Running the Cascade

The cascade runs in one process with NeMo/examples/asr/asr_streaming_inference/asr_streaming_infer.py: the unified model transcribes the audio in buffered streaming mode, and every finalized ASR segment is translated by an LLM served with vLLM. vLLM conflicts with the all / cu12 / cu13 extras, so install it into a separate environment of the NeMo Speech checkout:

cd Speech
UV_PROJECT_ENVIRONMENT=.venv-vllm uv sync --locked --python 3.13 --extra asr --extra vllm

The following command translates German speech into English with the setting of the first table row (0.56s chunk and right context, 5.6s left context):

.venv-vllm/bin/python examples/asr/asr_streaming_inference/asr_streaming_infer.py \
    --config-path=../conf/asr_streaming_inference \
    --config-name=buffered_rnnt.yaml \
    audio_file=<audio_file, directory or manifest> \
    output_filename=<output_json_file> \
    asr.model_name=nvidia/nemotron-labs-unified-asr-0.6b \
    lang=de \
    streaming.left_padding_size=5.6 \
    streaming.chunk_size=0.56 \
    streaming.right_padding_size=0.56 \
    asr_output_granularity=segment \
    enable_itn=False \
    endpointing.stop_history_eou=800 \
    enable_nmt=True \
    nmt.model_name=nvidia/Riva-Translate-4B-Instruct-v2 \
    nmt.source_language=German \
    nmt.target_language=English \
    nmt.waitk=-1 \
    nmt.device_id=1

• lang is the language ID of the speech, i.e. the source language of the translation pair; it selects the ASR language prompt.
• nmt.source_language and nmt.target_language are language names, not IDs. For Riva-Translate-4B-Instruct-v2, use Latin American Spanish and Brazilian Portuguese for the FLEURS Spanish and Portuguese data.
• nmt.waitk=-1 commits the longest common prefix of the current and previous translations; a positive value instead limits how many words the translation may lag behind the transcript. endpointing.stop_history_eou is the end-of-utterance window in milliseconds.
• The translation LLM runs on GPU nmt.device_id. To run both models on a single GPU, set nmt.device_id=0 and lower nmt.llm_params.gpu_memory_utilization (e.g. 0.6).

For English-to-X translation, set lang=en, nmt.source_language=English and the target language name.

References(s):

[1] Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization
[2] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
[3] Dynamic Chunk Convolution for Unified Streaming and Non-Streaming Conformer ASR
[4] Conformer: Convolution-augmented Transformer for Speech Recognition
[5] Granary: Speech Recognition and Translation Dataset in 25 European Languages
[6] NVIDIA Granary Dataset Card
[7] NVIDIA NeMo Framework
[8] NeMo@IWSLT 2026: Cascaded System for Simultaneous Speech Translation

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.