VoxCPM 1.5 — CoreML

CoreML conversion of VoxCPM 1.5 (800M params, 44.1kHz, bilingual EN/ZH text-to-speech with voice cloning).

Available in two precisions:

Constants (embeddings, projections) are shared between both variants: 299 MB in constants/.

Source

File Structure

├── constants/                         # Shared (299 MB)
│   ├── config.json
│   ├── embed_tokens.npy               # [73448, 1024] text embeddings (287 MB)
│   ├── enc_to_lm_proj_{w,b}.npy
│   ├── lm_to_dit_proj_{w,b}.npy
│   └── res_to_dit_proj_{w,b}.npy
├── fp16/                              # Float16 models (1.6 GB)
│   ├── audio_vae_encoder.mlpackage
│   ├── audio_vae_decoder.mlpackage
│   ├── feat_encoder.mlpackage
│   ├── base_lm_step.mlpackage
│   ├── residual_lm_step.mlpackage
│   └── locdit_step.mlpackage
└── int8/                              # INT8 models (783 MB each format)
    ├── audio_vae_encoder.mlpackage
    ├── audio_vae_encoder.mlmodelc
    ├── audio_vae_decoder.mlpackage
    ├── audio_vae_decoder.mlmodelc
    ├── feat_encoder.mlpackage
    ├── feat_encoder.mlmodelc
    ├── base_lm_step.mlpackage
    ├── base_lm_step.mlmodelc
    ├── residual_lm_step.mlpackage
    ├── residual_lm_step.mlmodelc
    ├── locdit_step.mlpackage
    └── locdit_step.mlmodelc

Models

6 CoreML models (.mlpackage) split for step-by-step autoregressive generation:

Architecture

VoxCPM 1.5 is a tokenizer-free diffusion autoregressive TTS built on MiniCPM-4 (0.5B LM backbone). It generates 44.1kHz audio at 6.25 Hz token rate using flow matching diffusion.

Total: ~800M parameters, 44.1kHz output, 6.25 Hz token rate (patch_size=4)

Conversion Validation

Per-component correlation (CoreML vs PyTorch FP32)

End-to-end verification (both variants produce identical results)

Generation Pipeline

1. Encode prompt audio: latent = audio_vae_encoder(pad_to_5s(prompt))
2. Reshape into patches: [1, 64, T] → [1, n_patches, 4, 64]
3. Encode patches: feat_emb = feat_encoder(each patch)
4. Project: feat_lm = enc_to_lm_proj(feat_emb)
5. Embed text: text_emb = embed_tokens[token_ids] * scale_emb
6. Combine: [text_emb, audio_start_token, feat_lm] → [1, seq_len, 1024]
7. Prefill: step through all tokens via base_lm_step + residual_lm_step
8. Loop (autoregressive):
   a. dit_hidden = lm_to_dit_proj(lm_hidden_fsq) + res_to_dit_proj(res_hidden)
   b. noise = randn(1, 64, 4)
   c. For t in 10 Euler steps (1.0 → 0.001):
        vel = locdit_step(noise, dit_hidden, prefix_cond, t)  # batch=2 for CFG
        noise = noise - vel * dt
   d. pred_feat = noise (after all steps)
   e. If stop_head predicts stop and step > min_len: break
   f. prefix_cond = pred_feat
   g. next_emb = enc_to_lm_proj(feat_encoder(pred_feat))
   h. lm_hidden, fsq, stop = base_lm_step(next_emb, pos, caches)
   i. res_hidden = residual_lm_step(fsq + next_emb, pos, caches)
9. Decode: audio = audio_vae_decoder(concat(all pred_feats))

Performance

Measured on Apple Silicon (macOS, CPU_AND_GPU compute units, INT8):

Usage

Requirements

• macOS 14+ on Apple Silicon
• Python 3.10+ with coremltools and numpy

Quick start

import coremltools as ct
import numpy as np

# Choose precision: "fp16" or "int8"
precision = "int8"

# Load models
vae_enc = ct.models.MLModel(f"{precision}/audio_vae_encoder.mlpackage")
vae_dec = ct.models.MLModel(f"{precision}/audio_vae_decoder.mlpackage")
feat_enc = ct.models.MLModel(f"{precision}/feat_encoder.mlpackage")
base_lm = ct.models.MLModel(f"{precision}/base_lm_step.mlpackage")
res_lm = ct.models.MLModel(f"{precision}/residual_lm_step.mlpackage")
locdit = ct.models.MLModel(f"{precision}/locdit_step.mlpackage")

# Load shared constants
embed_tokens = np.load("constants/embed_tokens.npy")
# ... see generate_coreml.py for full pipeline

Full generation script

See generate_coreml.py in the conversion repo for a complete zero-PyTorch generation pipeline:

python generate_coreml.py \
  --text "Hello, this is a test." \
  --prompt prompt.wav \
  --prompt-text "This is the prompt transcript." \
  --output output.wav

Conversion Details

Converted using coremltools with Float16 compute precision (compute_units=CPU_AND_GPU). INT8 variant post-quantized via ct.optimize.coreml.linear_quantize_weights() (linear symmetric).

Key conversion challenges solved:

1. GQA attention (16 query heads, 2 KV heads) — manual repeat_interleave expansion
2. Snake activations — replaced @torch.jit.script with simple module
3. In-place KV cache — functional scatter replacement
4. Euler solver direction — backward (1→0) with x = x - dt * v
5. Chinese tokenizer — mask_multichar_chinese_tokens wrapper for character splitting

See TRIALS.md for the full conversion log.