Nemotron 3 Diarization

Description

Nemotron 3 Diarization is an open-weight speaker diarization model designed to determine "who spoke when" in real-world audio. It supports both streaming and offline inference and handles up to eight speakers. Following Sortformer [1], the model resolves speaker permutation by ordering its output channels according to each speaker's first arrival in the input audio.

For streaming inference, the model adopts the Arrival-Order Speaker Cache (AOSC) and FIFO queue introduced in Streaming Sortformer [2]. The AOSC retains speaker information from earlier chunks to preserve speaker identities over time, while the FIFO queue provides recent frame context for each processing step.

A single checkpoint supports input buffer latency as low as 80 ms for latency-critical applications, although the lowest recommended configuration is 0.32 s; the offline-style configuration uses a 30.4 s input buffer. The output frame resolution is configurable in multiples of 10 ms. With chunked inference, the maximum audio duration is not limited.

This model is ready for commercial or non-commercial use.

Nemotron 3 Diarization HuggingFace Blog

Release Date

September 23, 2026

Live Action Demo Page

Hugging Face Spaces Demo: Nemotron-Diarization with Streaming ASR

What does max 8-speaker diarization mean ?

How to use this model

Coming soon...

---

Model Architecture

Architecture Type: Transformer.

Network Architecture:

• A 31-layer Transformer encoder with Rotary Positional Embeddings (RoPE).
• 10 ms Mel-spectrogram input features downsampled by a factor of eight through feature stacking, producing an 80 ms encoder frame rate.
• A Conv1D layer above the encoder that upsamples predictions to the 10 ms input-feature resolution.
• AOSC and a FIFO queue for streaming inference.

Number of model parameters: 100M (1.0 × 10⁸).

Input

Input Type: Audio.

Input Format: 16 kHz, single-channel audio in .wav, .flac, .opus, or .mp3 format.

Input Parameters: One-dimensional, 16 kHz, single-channel audio waveform.

Maximum Duration: Not limited when chunked inference is used.

Other Properties Related to Input: Input audio must be sampled at 16 kHz and is converted into 10 ms Mel-spectrogram features. Chunked inference supports recordings without a fixed maximum duration.

Output

Output Type: Other: Numerical tensor.

Output Format: Float Tensor.

Output Parameters: A two-dimensional tensor with shape [T, 8], where T is the number of output frames and each value is a per-speaker activity probability in the range [0, 1].

Output Frame Resolution: The default frame stride is 10 ms and can be configured to any multiple of 10 ms, such as 30 ms, 80 ms, or 240 ms.

Speaker Ordering: The eight speaker channels are ordered by the speakers' arrival time in the input audio.

Derived Output: Speaker activity probabilities can be converted into start time, end time, and a generic speaker label, for example ["speaker1", 0.51, 12.62].

Other Properties Related to Output: The output frame resolution is configurable in multiples of 10 ms. Speaker channels are ordered by first arrival, and probabilities can be postprocessed into generic speaker labels with start and end timestamps.

---

Software Integration

Runtime Engine(s): NeMo Framework v3.0.

Supported Hardware Microarchitecture Compatibility

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems.

Ampere NVIDIA GPUs:

• GeForce RTX 3090 Ti, RTX 3090, RTX 3080 Ti, RTX 3080, RTX 3070 Ti, RTX 3070, RTX 3060 Ti, RTX 3060, and RTX 3050.
• NVIDIA RTX A6000, RTX A5500, RTX A5000, RTX A4500, RTX A4000, RTX A2000, RTX A1000, and RTX A400.
• NVIDIA A100 PCIe, A100 SXM, A30, A40, A16, A10, and A2.

Ampere NVIDIA Workstations: NVIDIA DGX Station A100.

Ada Lovelace NVIDIA GPUs:

• GeForce RTX 4090, RTX 4080, RTX 4070 Ti, RTX 4070, RTX 4060 Ti, RTX 4060, and RTX 4050.
• NVIDIA RTX 6000 Ada, RTX 5000 Ada, RTX 4500 Ada, RTX 4000 Ada, RTX 4000 SFF Ada, and RTX 2000 Ada.
• NVIDIA L4, L40, and L40S.

Blackwell NVIDIA GPUs:

• GeForce RTX 5090, RTX 5080, RTX 5070 Ti, RTX 5070, RTX 5060 Ti, RTX 5060, and RTX 5050.
• RTX PRO 6000 Blackwell Workstation Edition, RTX PRO 6000 Blackwell Max-Q Workstation Edition, RTX PRO 5000 Blackwell, RTX PRO 4500 Blackwell Workstation Edition, RTX PRO 4000 Blackwell, RTX PRO 4000 Blackwell SFF Edition, and RTX PRO 2000 Blackwell.
• RTX PRO 6000 Blackwell Server Edition and RTX PRO 4500 Blackwell Server Edition.
• NVIDIA B200, B300, GB200, and GB300.

Blackwell NVIDIA Workstations: NVIDIA DGX Spark and NVIDIA DGX Station.

Hopper NVIDIA GPUs: NVIDIA H100 PCIe, H100 SXM, H100 NVL, H200 SXM, H200 NVL, and GH200.

Preferred/Supported Operating System(s):

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

---

Model Version

Nemotron 3 Diarization (General Access).

The model can be integrated into conversational AI systems through NeMo Framework inference pipelines to produce per-speaker activity probabilities or postprocessed generic speaker labels and timestamps.

Training and Evaluation Datasets

Training Datasets

The model was trained on a combination of real conversations and multi-talker audio mixtures with 1-8 speakers simulated using the FastMSS toolkit [3].

Real conversations: Total duration is around 10,000 hours.

Data used to simulate multi-talker audio mixtures: Total duration of single-speaker audio recordings is around 28,000 hours.

Multi-talker audio mixtures used in training:

• Librispeech: 6,694 hours
• AMI: 6,743 hours
• AliMeeting: 6,745 hours
• Fisher English: 6,755 hours
• David AI English: 36,458 hours
• David AI Multilingual: 19,216 hours

Properties: Approximately 10,000 hours of real conversations plus 82,611 hours of simulated multi-talker audio mixtures. The data modality is audio and includes conversational speech, telephone calls, meetings, podcasts, noise augmentation, and synthetic mixtures. Languages include English, Mandarin, Hindi, Kannada, Telugu, Bengali, and other languages represented in the multilingual sources. Voice recordings may constitute personal data.

Evaluation Dataset

Properties: 901 condition-specific audio recordings comprising real-world multilingual conversational speech captured under telephone, meeting, near-field, far-field, and multi-microphone conditions. The data may contain personal data in the form of voice recordings. Languages include English, Mandarin, and other languages represented in the multilingual benchmarks.

---

Training

The model training was initialized with a Transformer-based NEST [5] SSL checkpoint. Training was performed on 8 nodes of 8×NVIDIA A100-SXM4-80GB GPUs in two stages.

1. Offline training on simulated data only using the example script and base config.
2. Streaming fine-tuning on a combination of real conversations and simulated mixtures using the example script and base config.

Performance Evaluation

[!IMPORTANT]

Use the published reference labels to reproduce these results!

The DER scores reported in this model card were computed using the exact reference annotations identified in the Labels column above. Reference RTTMs are part of the evaluation protocol: changing the reference labels changes the measured result.

We use forced-alignment based reference labels for AMI, AliMeeting and NOTSOFAR1 because the original segment-level annotations were created primarily for transcription rather than frame-accurate diarization evaluation. These annotations may label substantial within-segment silence as speech, thereby overestimating reference speaker activity. When used for DER scoring, they can inflate missed-speech error by penalizing a diarization system for correctly predicting non-speech during those intervals. Forced alignment provides more precise speech boundaries and therefore a more appropriate and interpretable reference for frame-level diarization evaluation. Please refer to [4] for a detailed discussion of this annotation issue and the forced-alignment methodology.

The Forced alignment links above point to the public repositories containing the reference RTTM files used for evaluation.

Results obtained using different reference labels constitute a different evaluation protocol and are not directly comparable with the numbers reported here. Before reporting a reproduction discrepancy, score the same model outputs using the linked reference RTTMs, the listed dataset split, and the collar and overlap settings specified in the Metrics section.

Baseline

nvidia/diar_streaming_sortformer_4spk-v2.1

Baseline model's latency configurations:

Metrics

Diarization Error Rate (DER): The primary metric for diarization performance, consisting of false alarm (FA), missed speech (Miss), and speaker confusion (Conf).

• All evaluations include overlapping speech.
• Collar tolerance is 0 s for DIHARD III Eval, AliMeeting Test, AMI Test and NOTSOFAR1 Eval.
• Collar tolerance is 0.25 s for CALLHOME-part2.

Speaker Counting Accuracy (SCA): 1 when the predicted and ground-truth speaker counts are equal; otherwise 0. This metric does not capture the magnitude of a counting error.

Speaker Counting Mean Absolute Error (MAE): |predicted speaker count - ground-truth speaker count|. This metric is more informative than SCA because it reflects magnitude of speaker counting error.

Real-Time Factor Speedup (RTFx): total audio duration / total processing time.

Performance Evaluation Results

Results reported below were obtained using the NeMo example script e2e_diarize_speech.py.

DIHARD III

CALLHOME-Part2

AliMeeting Test Near

AliMeeting Test Far

AMI Test MHM

AMI Test SDM

NOTSOFAR1 MHM

NOTSOFAR1 SC

Inference Speed Evaluation

Acceleration Engine: PyTorch backend through the NeMo Framework, with and without torch.compile().

Test Hardware: NVIDIA Blackwell RTX PRO 5000.

Precision: BF16.

---

License/Terms of Use

Use of this model is governed by the OpenMDW License Agreement, version 1.1.

Deployment Geography

Global

Use Case

Nemotron 3 Diarization is intended for speaker diarization in live or recorded conversational audio, including meetings, calls, podcasts, and speech-recognition pipelines that need generic speaker labels and speaker timestamps.

References

[1] Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems

[2] Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering

[3] Mind the Gap: Impact of Synthetic Conversational Data on Multi-Talker ASR and Speaker Diarization

[4] Can We Really Repurpose Multi-Speaker ASR Corpus for Speaker Diarization?

[5] NEST: Self-supervised Fast Conformer as All-purpose Seasoning to Speech Processing Tasks

---

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and has established policies and practices to enable development for a wide array of AI applications. Developers should evaluate the model with use-case-specific data and ensure that the complete system meets the requirements of the relevant industry and deployment context.

For more detailed information, see the Bias, Explainability, Safety & Security, and Privacy subcards.

Please report model quality, risk, security vulnerabilities, or NVIDIA AI concerns through the NVIDIA security reporting portal.