PhoneLLM Alpha 2: a blazingly fast, open-weights text model designed to be integrated into voice agent pipelines.

This new version is available in two variants: Mini for speed, and Max for accuracy.

Check out our blog post to find out more about the improvements we made, and how we trained the models:

https://www.daily.co/blog/phonellm-alpha-2/

---

Pipecat PhoneLLM Max Alpha 2 (NVFP4)

The NVFP4 quantization of PhoneLLM Max Alpha 2, a full-parameter supervised fine-tune of the language backbone of NVIDIA Nemotron 3.5 Super VL 120B-A12B for phone voice agents with tool calling.

• Architecture: text-only NemotronHForCausalLM (the vision encoder and projector were removed before training)
• Quantization: NVIDIA Model Optimizer (ModelOpt) post-training quantization, calibrated on 512 examples from the model's own training data. Routed and shared experts are NVFP4; attention, Mamba projections and lm_head are FP8; embeddings, the router, convolutions and most of the MTP block stay bf16. FP8 KV-cache scales are included. ~73 GB, so it fits on one GPU; NVFP4 needs a Blackwell GPU (B200, B300, RTX PRO 6000) for native FP4 kernels.
• Requirements: vLLM >= 0.30 (it reads the ModelOpt quantization config from config.json)
• Thinking is ON: the model emits one short sentence of reasoning before every response — we refer to this as "microthinking". The model has not been trained to run with thinking disabled.
  • The bundled chat_template.jinja is the stock Nemotron template (thinking on by default).
  • Serve with --reasoning-parser nemotron_v3 and send chat_template_kwargs: {"enable_thinking": true, "truncate_history_thinking": false} on every request — the parser reads the mode from the request, so a request that omits the kwargs has its reply filed under reasoning.
  • The thought comes back in the message's reasoning field; to keep earlier thoughts in the prompt (as in training) re-send each assistant message with its reasoning.
• Recommended settings: temperature=0.

Serving with vLLM

Smoke-tested recipe (vLLM 0.30, one B200); throughput has not been benchmarked yet.

vllm serve pipecat-ai/phonellm-max-alpha-2-nvfp4 \
  --served-model-name phonellm-max-alpha-2-nvfp4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --kv-cache-dtype bfloat16 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-ssm-cache-dtype float32 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

• Keep --kv-cache-dtype bfloat16: the checkpoint carries FP8 KV-cache scales, so without the flag vLLM silently uses an FP8 KV cache. FP8 KV holds roughly 40 % more tokens, but on PhoneLLM Alpha 1 NVFP4 it cost 0.5 PhoneBench points (bf16 KV: within 0.04 of the bf16 model). Switch to --kv-cache-dtype fp8 only if you need the capacity.
• Do not add --enforce-eager: CUDA graphs are ~7x faster decode on this model.
• If CUDA-graph capture runs out of memory, use --max-num-seqs 64 --max-cudagraph-capture-size 64 rather than eager mode.
• To make thinking unconditional on a shared endpoint, rewrite requests server-side (we run a small ASGI middleware through vLLM's --middleware) rather than editing the template: the reasoning parser must agree with the template on the mode.
• The weights include the base model's multi-token-prediction (MTP) block; the recipe above does not use it.

License

PhoneLLM Max Alpha 2 is released under the OpenMDW License Agreement, version 1.1, the same licence as its base model, NVIDIA Nemotron 3.5 Super VL 120B-A12B. If you redistribute this model or your own derivatives of it, you must retain (1) a copy of the OpenMDW License Agreement and (2) the copyright notices and other notices of origin that apply to your distribution. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.