Nemotron 3.5 Transcribe 30B-A4B — CMF

NVIDIA Nemotron 3.5 Transcribe (34 B total, 4.3 B active: a transformer ASR encoder and a Sortformer speaker encoder feeding a hybrid Mamba-2 / attention / MoE language model, 40 languages) packaged as single CMF files. They run with cortiq, a Rust inference engine with no Python and no ML framework: the whole pipeline — log-mel front end, both audio towers, speaker cache, decoder, word timestamps — runs on the GPU through Vulkan/DX12/Metal (wgpu), or on the CPU.

English · Русский · 中文

Quick start

cargo install cortiq-cli
hf download infosave/Nemotron-3.5-Transcribe-30B-A4B-cmf nemotron-3.5-transcribe-30b-a4b-q4tp.cmf --local-dir .
cortiq transcribe nemotron-3.5-transcribe-30b-a4b-q4tp.cmf meeting.wav

Use cortiq 0.8.47 or later; prebuilt binaries are on the releases page. Input is WAV (PCM 8/16/24/32-bit or float, any sample rate, mono or stereo). Several files in one call share one model load and print <file>: <text> lines.

Files

Both files also carry the multi-token-prediction layers, the word-timestamp head with its alignment vocabulary, and each MoE layer's experts ordered by use.

WER: 73 recordings of hf-internal-testing/librispeech_asr_dummy (1150 words), default prompt, greedy, lower-cased and punctuation stripped before scoring. With the Whisper English normalizer of the Open ASR Leaderboard the same set scores 1.71 % (q8_2f) and 1.88 % (q4tp).

LibriSpeech test-clean (all 2620 recordings, 52,576 words, leaderboard normalizer): q4tp 1.44 %; NVIDIA reports 1.09 % for the bf16 model.

The q4tp experts are GPTQ-encoded against the expert inputs of 11 k tokens (73 recordings and their transcripts); the 79 of 2944 experts routed fewer than 32 calibration rows keep round-to-nearest encoding.

Parity with the reference implementation (q8_2f file, teacher forcing on the Transformers transcripts, 7 recordings up to 97.6 s): ASR encoder output relative error 5e-7 – 7e-6, speaker activities identical, prompt token ids identical, next-token argmax agreement 96–100 %. Word timestamps: the timestamp head's argmax agrees with Transformers on 99.8–100 % of frames and the aligned word times are identical.

Performance

cortiq transcribe --stats, one recording at a time:

q8_2f on the CPU alone (AMD EPYC 7H12, 31 cores): 29.4 s recording in 24.5 s (RTF 0.83).

On a discrete GPU the decoder runs as a resident frame: the whole token in one pass of fused kernels; the prompt is prefilled 64 positions per pass. With the q4tp file the decode is speculative (below). When the routed experts do not fit the card (the q8_2f file on 24 GB), each MoE layer keeps its most used experts on the card (75 of 128 on a 3090), the others are computed on the CPU from the mapped file — which reads them from the disk when RAM is short — and experts that keep being picked are moved onto the card. CMF_NEMO_EXPERT_GB=<GB> sets the card budget for experts.

On the M4 the file runs from the memory map (the native Metal backend, the default on macOS); the first recording of a process pays the one-time GPU setup.

Usage

# verbatim transcript (the model's default instruction)
cortiq transcribe model.cmf talk.wav
# word timestamps: one "start end word" line per word (seconds)
cortiq transcribe model.cmf talk.wav --timestamps
# clean written transcript
cortiq transcribe model.cmf talk.wav --prompt "Audio transcription. Language: as spoken. Wording: omit fillers and accidental repetitions; keep the remaining spoken wording unchanged. Formatting: punctuation and capitalization. Output: transcript only."
# who said what (with --timestamps every word also gets its speaker)
cortiq transcribe model.cmf meeting.wav --prompt "Transcribe this audio with punctuation and capitalization. Include all spoken words and verbal fillers. Write numbers in their standard written form. Put speaker tags <spk:0> <spk:1> <spk:2> <spk:3> on the transcription."
# speech translation
cortiq transcribe model.cmf talk.wav --prompt "Please translate the given speech to German."

Decoding is greedy with reasoning off, as NVIDIA recommends for transcription and translation. --max-new-tokens (default 16384) caps the output. Recordings up to an hour fit the 65,536-token context; on unified memory (Apple) the KV cache defaults to 20,480 positions (about 27 minutes of audio), CMF_NEMO_CTX=<positions> changes it.

Word timestamps follow the reference aligner: the timestamp head classifies every 40 ms frame of the ASR encoder states into the pieces of its alignment vocabulary, the transcript's words are force-aligned on them (CTC), and in a speaker-tagged transcript each tag's words are aligned on the frames where its diarization speaker is active and snapped to that speaker's segments. The head runs on the CPU (about 0.03 × the audio duration on 31 cores). Chinese and Japanese are aligned per character; Thai is not supported.

When every expert fits the card (the q4tp file on 24 GB) the decode is speculative with the model's multi-token-prediction layer: each pass verifies the next token and the layer's draft of the one after it, and keeps the draft when the model agrees — the output is the plain greedy decode's. 96 % of the drafts are kept on the 73 recordings; the decode runs 1.1–2× faster. CMF_NEMO_MTP=0 turns it off.

Verify

sha256sum nemotron-3.5-transcribe-30b-a4b-q4tp.cmf nemotron-3.5-transcribe-30b-a4b-q8_2f.cmf

License

The model is NVIDIA's, under the OpenMDW License Agreement 1.1; THIRD_PARTY_NOTICES.md and licenses/ are NVIDIA's notices, kept as the license requires. The CMF packaging adds no terms.

---

Документация на русском

CMF-версия NVIDIA Nemotron 3.5 Transcribe (34 B, 4.3 B активных, 40 языков) для движка cortiq на Rust: мел-фронтенд, ASR-энкодер, энкодер спикеров, кэш спикеров, декодер Mamba-2/MoE и пословные таймкоды целиком на GPU (Vulkan/DX12/Metal) или CPU.

cargo install cortiq-cli
hf download infosave/Nemotron-3.5-Transcribe-30B-A4B-cmf nemotron-3.5-transcribe-30b-a4b-q4tp.cmf --local-dir .
cortiq transcribe nemotron-3.5-transcribe-30b-a4b-q4tp.cmf запись.wav
cortiq transcribe nemotron-3.5-transcribe-30b-a4b-q4tp.cmf запись.wav --timestamps

Нужен cortiq 0.8.47+. Файлы: q4tp (19.86 ГБ, эксперты 4 бит GPTQ, WER 2.26 %, LibriSpeech test-clean 1.44 %, для GPU 24 ГБ — по умолчанию) и q8_2f (34.59 ГБ, WER 2.09 %; на GPU 24 ГБ часть экспертов считается на CPU из отображённого файла). RTX 3090: запись 29.4 с за 1.82 с (RTF 0.062), декод 182 ток/с (спекулятивный, слоем MTP); q8_2f на той же карте — RTF 0.19–0.25; Mac mini M4 24 ГБ (Metal): RTF 0.86. --timestamps — время начала и конца каждого слова (и спикер в размеченной по спикерам расшифровке); --prompt — свой запрос (чистый текст, метки спикеров <spk:N>, перевод); CMF_NEMO_MTP=0 выключает спекулятивный декод. Лицензия — OpenMDW-1.1 от NVIDIA.

中文文档

NVIDIA Nemotron 3.5 Transcribe （总参数 34 B，激活 4.3 B，支持 40 种语言）的 CMF 单文件版本，由 Rust 推理引擎 cortiq 运行：梅尔前端、两个音频编码器、说话人缓存、Mamba-2/MoE 解码器和逐词时间戳 全部在 GPU（Vulkan/DX12/Metal）或 CPU 上执行。

cargo install cortiq-cli
hf download infosave/Nemotron-3.5-Transcribe-30B-A4B-cmf nemotron-3.5-transcribe-30b-a4b-q4tp.cmf --local-dir .
cortiq transcribe nemotron-3.5-transcribe-30b-a4b-q4tp.cmf audio.wav
cortiq transcribe nemotron-3.5-transcribe-30b-a4b-q4tp.cmf audio.wav --timestamps

需要 cortiq 0.8.47 及以上。q4tp（19.86 GB，专家 4 位 GPTQ，WER 2.26 %，LibriSpeech test-clean 1.44 %，默认，适合 24 GB 显卡）；q8_2f（34.59 GB，WER 2.09 %；在 24 GB 显卡上 部分专家由 CPU 从映射文件计算）。RTX 3090 上 29.4 秒录音用时 1.82 秒（RTF 0.062），解码 182 token/s（MTP 推测解码）；q8_2f 在同一显卡上 RTF 0.19–0.25；Mac mini M4（24 GB，Metal）RTF 0.86。 --timestamps 输出每个词的起止时间（按说话人标注时还有说话人）。许可证：NVIDIA OpenMDW-1.1。