NemotronLabs VoiceChat 11B: GPTQ 3-bit LLM, MLX

▶ Run it locally in LLMTray

LLMTray is a free, native macOS app for local AI on Apple Silicon. It does chat, images, music, agents, voice and an OpenAI-compatible API. Point it at this model; nothing leaves your Mac.

This is an MLX build of NVIDIA's full-duplex speech-to-speech model, nvidia/NVIDIA-NemotronLabs-VoiceChat-11B. The model listens, transcribes, answers with text and speaks on one continuous 80 ms timeline.

The language model inside is Nemotron-Nano-9B-v2, a hybrid of Mamba-2 and attention with 56 layers. It is GPTQ-quantized to 3 bits (group size 64), calibrated on real duplex conversations. The speech parts stay in bf16: the FastConformer encoder, the TTS and the codec.

The goal was an 80 ms frame in close to 80 ms on a base M5 MacBook, for LLMTray's Voice Lab.

• 7.6 GB of weights; 9.1 GB MLX peak memory in a session (10.6 GB for the 4-bit build).
• Every answer is correct on our 20 spoken test questions. Its reply WER is the lowest of the variants we measured.
• The LLM step drops from 62 ms to about 40 ms per frame compared with the 4-bit build. The LLM step is limited by memory bandwidth, so fewer bits read means a faster step.

Recipe

Calibration. The calibration set is 256 real duplex conversations. Each spoken clip was streamed through the bf16 model's own streaming session: system-prompt prefill, then the user's speech, then silence until the reply ended. The LLM's fused per-frame input was captured at every step. GPTQ therefore sees the same inputs as the model does at inference: speech embeddings, user-transcript tokens and the model's own replies.

Measurements (base M5, 26 GB, 2026-09-28)

The test set is 20 spoken questions, each with keywords the answer must contain. The model's spoken reply is transcribed by an independent Whisper. Timings are milliseconds per 80 ms duplex frame, averaged over every frame of every question.

Two things together give the 185 → 88 ms:

• This checkpoint accounts for the LLM column.
• Our mlx-audio fork accounts for the other columns:
  • perception runs in bf16, with the conformer compiled for its steady state;
  • the TTS mixture head computes only the sampled mixture;
  • TTS code generation and the codec step are compiled.

On a base M5 the model is not quite real-time yet: about 1.1× slower than real time (95th-percentile frame 99 ms). On M5 Pro or Max chips, with more memory bandwidth, it should be faster, but we haven't measured them. Background GPU load (the UI, a browser) shifts these numbers by about ±5 ms between runs.

Usage

LLMTray: open Voice Lab, which is opt-in in Settings, and pick this model. LLMTray unloads the chat model while you talk.

Python: use our mlx-audio fork, which has the speedups above:

pip install "mlx-audio[stt] @ git+https://github.com/ipsupport-llc/mlx-audio.git@llmtray"

from mlx_audio.sts import load

model = load("roman220220/NemotronLabs-VoiceChat-11B-gptq-mlx-3bit")

# Offline: a WAV in, text + speech out
output = model.generate("input.wav")
print(output.text)

# Full duplex: keep one session and feed 16 kHz mono PCM as it arrives
session = model.create_duplex_session()
for chunk in microphone_chunks:
    for event in session.push_audio(chunk, sample_rate=16_000):
        if event.kind == "assistant_text_delta":
            print(event.delta, end="", flush=True)
        elif event.kind == "audio":
            play(event.samples, event.sample_rate)  # 22.05 kHz
session.flush()

Input is 16 kHz mono. Output is 22,050 Hz mono in the model's built-in voice. The model is English only. The checkpoint uses the same config layout as the mlx-community conversions (mlx_runtime_config_version: 2) and ships its tokenizer, so a downloaded copy loads with HF_HUB_OFFLINE=1.

Method / code

• Quantization pipeline, calibration capture, evaluation and the full results: rromenskyi/quant-ternary/voicechat-quant (calib_capture.py, gptq_llm.py, splice_llm.py, vc_eval.py, docs/FINDINGS.md).
• Model code: ipsupport-llc/mlx-audio (mlx_audio/sts/models/nemotron_voicechat).
• Source conversion: mlx-community/NemotronLabs-VoiceChat-11B-bf16 and -4bit, built from NVIDIA's checkpoint at source revision c5d3b70183b6bb9d7553590e111b05685049751c.

Intended use and limitations

This model inherits NVIDIA's intended use, limitations and responsible-AI notes. See the base model card and the notices shipped in this repo (bias.md, explainability.md, privacy.md, safety.md).

Quantizing to 3 bits changes the weights. We checked quality only on the English test set described above. Validate it for your own use case.

The IPSupport local-AI stack

Local-first AI tools for macOS by IPSupport — nothing leaves your Mac.

• LLMTray is your local AI workstation for macOS. Chat with local LLMs, generate and edit images, make music, run agents, and serve an OpenAI-compatible API. It downloads models from Hugging Face in-app, with per-model profiles.
• IPSupport Code is your AI coding agent for real repositories: analyze, fix, test, report.

LLMTray runs this model in Voice Lab for full-duplex local voice conversation.

License

This model is licensed under the OpenMDW License Agreement, version 1.1, the same license as the base model. See LICENSE, which keeps NVIDIA's copyright notice.

Modified from nvidia/NVIDIA-NemotronLabs-VoiceChat-11B:

• converted to MLX;
• the language model (Nemotron-Nano-9B-v2) was GPTQ-quantized to 3 bits at group size 64;
• the LM head, function head and embeddings were quantized to 4 bits;
• perception, TTS and codec were left in bf16.

The weights and configuration files in this repo are therefore modified versions of the original, not the original files.

NemotronLabs VoiceChat is © 2026 NVIDIA CORPORATION & AFFILIATES and is released under OpenMDW 1.1. NVIDIA's model-card notices (bias.md, explainability.md, privacy.md, safety.md) are included unchanged.