Nemotron 3.5 Transcribe Lite: Core ML + Core AI

Unofficial community port, not affiliated with or endorsed by NVIDIA or Apple. "Lite" is our name for this setup, not an NVIDIA model or release. It is the audio side of NVIDIA Nemotron 3.5 Transcribe, without its 30B language model, converted for Apple silicon:

• the ~600M-parameter ASR encoder and the CTC timestamp head (words with 40 ms start times), as Core ML and Core AI models
• run next to Nemotron-3-Diarization (who spoke when). The diarizer is not in this repo: FluidAudio downloads its Core ML port from FluidInference/nemotron-3-diarization-coreml

On an M5 Pro MacBook Pro, a 49.5-minute, 3-person meeting becomes a speaker-labeled transcript with word start times in about 6 s with the models preloaded and warm (~500x real time): the encoder and CTC head run on the GPU while the diarizer runs on the Neural Engine at the same time. Model loading, warm-up and the diarizer's one-time Neural Engine compile (~80 s) are not included (see Meeting demo).

This is not the full model. Without the LLM the transcript is much rougher (see Speed and accuracy). The encoder weights here are NVIDIA's, unchanged apart from the format conversion; the CTC head was trained by NVIDIA as a timestamp aligner for the LLM's output, and here it is used directly as a greedy CTC recognizer.

Files

Inputs and outputs (enc_len is (1,) int32; token_ids and token_logprob give one token per 40 ms):

• lite_encoder_w48: mel (1, 128, 4800) fp32 + enc_len → states (1, 1280, 600) fp16
• lite_ctc_head_w48: states (1, 1280, 600) + enc_len → token_ids, token_logprob (1, 1200)
• lite_ctc_fp16, _w8, _p4 and the .aimodel: mel (1, 128, 3000) + enc_len → token_ids, token_logprob (1, 750)

All Core ML models are fp16 ML Programs (macOS 15+) unless noted. w48 = 4800 mel frames: NeMo's long-form layout of 40 s cores with 4 s of context on each side; the demo keeps only each window's core frames. In lite_ctc_p4 the CTC output projection (23,863 x 1280) stays fp16, because 23,863 rows do not split into groups of 16.

The .mlmodelc folders, assets/*.bin and assets/vocab.json are for FluidAudio (Swift); FluidAudio support: FluidAudio #1069 (nemotron35-transcribe).

Meeting demo

Needs Python 3.10 to 3.13 (coremltools 9.0 has no Python 3.14 wheel). The download fetches only what the demo uses, about 1.4 GB of this repo (FluidInference/nemotron-3.5-transcribe-lite-coreml); leave out the --include flags to get everything.

python3.13 -m venv .venv && . .venv/bin/activate      # any Python 3.10-3.13
pip install -U huggingface_hub                        # provides the `hf` CLI
hf download FluidInference/nemotron-3.5-transcribe-lite-coreml --local-dir nemotron-lite \
  --include "coreml/lite_*_w48.mlpackage/*" --include "assets/*" --include "demo/*" --include requirements.txt

git clone https://github.com/FluidInference/FluidAudio && (cd FluidAudio && swift build -c release)
export FLUIDAUDIO_CLI=$PWD/FluidAudio/.build/release/fluidaudiocli  # diarizer (~200 MB) is downloaded on first use

cd nemotron-lite && pip install -r requirements.txt
python demo/lite_meeting.py meeting.wav --verbose          # 16 kHz audio; ffmpeg -i in -ac 1 -ar 16000 meeting.wav
python demo/lite_meeting.py meeting.wav --out t.txt --json t.json   # transcript with <spk:N> tags; word start times

--verbose logs every window; --wait loads and warms all three models, then starts the clock when you press Enter. bash demo/demo_tmux.sh meeting.wav opens a side-by-side view (needs brew install tmux macmon): CPU, GPU and Neural Engine power on the left (demo/sidemon.py), the run on the right. Run the shell scripts with bash: Hugging Face downloads do not keep the executable bit.

Pipeline: mel + per-feature normalization (NumPy) → encoder windows (GPU) → CTC head windows (GPU) → greedy CTC words, each stamped with its first piece's 40 ms frame (start time only). The diarizer starts at t=0 in its own process on the Neural Engine; each word takes the speaker active at its time (nearest active 80 ms frame within 1 s). Diarizer slots with under 2 s of speech in the whole recording are not counted as speakers.

The first Neural Engine run compiles the diarizer (~80 s on an M5 Pro). macOS keeps the compiled copy only for a while, so the demo warms it on a 10 s clip before the clock starts.

Speed and accuracy

M5 Pro (24 GB), macOS 27, models loaded and warmed before timing.

Meeting: AMI Meeting Corpus EN2002c (49.5 min, mixed headset audio, 3 speakers), scored against the AMI manual transcript with the Whisper English normalizer.

For comparison, NVIDIA reports 10.29% cpWER (individual headsets) and 17.34% (single distant mic) on AMI for the full model with its LLM. Those figures are for the AMI test set (full recordings), scored with NVIDIA's own pipeline. Ours is one meeting on the mixed-headset channel, scored with our own jiwer + Whisper-normalizer cpWER, so the comparison is only indicative.

LibriSpeech test-clean, first 400 utterances (7 of 40 speakers), greedy CTC, 30 s windows (shorter clips padded to 30 s), speed = audio duration / time inside the model call:

The 400-utterance subset is easier than the full set: on all 2,620 test-clean utterances the PyTorch encoder + CTC head scores 5.50%. NVIDIA reports 1.09% for the full model with its LLM. Core ML fp16 and Core AI stay within 0.03 points of PyTorch's WER (at most 2 word errors apart out of 9,004 words; Core ML fp16 on the GPU has the same error count). Neural Engine speeds here use the standard tensor layout, not Apple's ANE-optimized layout.

Only English was evaluated.

Core AI

coreai/lite_ctc_gpu.aimodel needs Apple's Core AI runtime (coreai-core==1.0.0b3, beta, Python 3.12) on macOS 26+ (its only Mac wheels are macosx_26_0_arm64). Loading and calling are async in Python:

import asyncio, numpy as np, coreai.runtime as rt

async def main():
    kind = rt.ComputeUnitKind.gpu  # or rt.ComputeUnitKind.neural_engine
    opts = rt.SpecializationOptions.from_preferred_compute_unit_kind(kind())
    m = await rt.AIModel.load("coreai/lite_ctc_gpu.aimodel", opts)
    fn = m.load_function(list(m.function_names)[0])
    mel = np.zeros((1, 128, 3000), np.float32)  # normalized mel, see demo/frontend.py
    r = fn({"mel": rt.NDArray(mel), "enc_len": rt.NDArray(np.array([375], np.int32))})
    r = await r if hasattr(r, "__await__") else r
    print(r["token_ids"].numpy().shape)  # (1, 750): one token per 40 ms

asyncio.run(main())

coreai/eval_aimodel.py is the LibriSpeech evaluation above.

How it was made

1. scripts/extract_lite.py: reads the index of NVIDIA's 67.5 GB model.safetensors and downloads only the 893 perception.* and ctc_timestamp.* tensors (1.67 GB) with HTTP Range requests → ckpt/lite.safetensors at the repo root, where the other scripts read it.
2. scripts/export_assets.py: mel window, filterbank and CTC tokenizer → assets/.
3. scripts/lite_model.py + scripts/attn_patches.py: the encoder + CTC head in PyTorch via NeMo, with FlexAttention replaced by an exact masked-matmul attention so it traces.
4. scripts/convert_coreml.py (lite_ctc_*), scripts/convert_w48.py (the demo's encoder + head), coreai/export_aimodel.py (Core AI).
5. scripts/eval_librispeech.py, scripts/eval_coreml.py, coreai/eval_aimodel.py: the LibriSpeech numbers above. The AMI meeting WER/cpWER scorer is not included.

Run the scripts from the repo root. The demo needs none of the following.

• NeMo Speech must come from git: PyPI nemo-toolkit 3.0.0 lacks the RoPE encoder and the CTC timestamp head. pip install -r scripts/requirements-convert.txt installs the toolchain the Core ML models were built with (Python 3.12; NeMo Speech commit 16b0ff2584, torch 2.7.0, coremltools 9.0, numpy 1.26.4: coremltools 9.0 fails to convert under numpy 2.5).
• coreai/export_aimodel.py used a separate env: Python 3.12, torch==2.9.0, coreai-core==1.0.0b3, coreai-torch==0.4.3, coreai-opt==0.3.0 and the same NeMo commit. Its parity gate needs PyTorch reference outputs (coreai/oracle.npz) that are not shipped; without them the gate is skipped.
• The parity check in scripts/convert_coreml.py and the eval scripts read LibriSpeech test-clean (CC BY 4.0) from data/LibriSpeech/test-clean: mkdir -p data && curl -L https://www.openslr.org/resources/12/test-clean.tar.gz | tar xz -C data

Changelog

License

The model materials are NVIDIA's, released under the OpenMDW License Agreement 1.1 (copy in LICENSE). This repository redistributes them converted to Core ML and Core AI formats, including 8-bit and 4-bit compressed variants. Unofficial port, not affiliated with or endorsed by NVIDIA or Apple.

scripts/attn_patches.py is adapted from NVIDIA NeMo Speech (Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES) and is licensed under the Apache License 2.0 (copy in licenses/NeMo-LICENSE.txt).

Meeting numbers use the AMI Meeting Corpus (CC BY 4.0); no AMI audio is included here.