The text decoder of openai/whisper-large-v3-turbo, converted to one stateful Core ML model that runs on the Apple Neural Engine. It was made for Speakey, a dictation app for the Mac, which downloads it by itself. The weights are OpenAI's, in fp16, and were not retrained.
It replaces only the decoder. The audio encoder, mel spectrogram and tokenizer stay Argmax's WhisperKit build (openai_whisper-large-v3-v20240930_turbo), and the model plugs into WhisperKit as its TextDecoding.
An int8 build for WhisperKit's smaller 547 MB variant is also available.
WhisperKit's decoder takes one token per call, so a vocabulary prompt goes through it a token at a time. This model keeps the KV cache in Core ML state and reads up to 64 tokens per call.
| M5 Max, warm | WhisperKit decoder | This decoder |
|---|---|---|
| Per token | 6.1 ms | 3.0 ms |
| 120-token prompt | 708 ms | 16 ms |
| Energy per decode | 1.22 J | 0.44 J |
| A whole take in Speakey, 12 Indonesian and English takes, p50 | ~390 ms | ~345 ms |
The words match WhisperKit's on the same audio. Both decoders are about the same distance from a full-precision PyTorch reference.
| File | Size | What it is |
|---|---|---|
decoder.mlmodelc |
343 MB | The decoder: 4 layers, fp16, functions cross, main and step sharing one MLState |
embed_tokens.f16 |
133 MB | Token embeddings, raw float16, 51,866 x 1280 |
embed_positions.f16 |
1.1 MB | Position embeddings, raw float16, 448 x 1280 |
speakey.json |
2 KB | Shapes, special token ids and file sizes |
LICENSE |
MIT, from OpenAI |
Requires macOS 15 or later (Core ML state) on Apple silicon.
cross takes the encoder output enc (1, 1280, 1, 1500), writes each layer's cross-attention keys and values into the state, and clears the self-attention cache. Call it once per take.main reads 64 rows: x (1, 1280, 1, 64) is token plus position embeddings, channels first. place (1, 1, 224, 64) and placeT are one-hot matrices that say which cache row each input row writes, mask (1, 1, 64, 224) is additive (0 or -1e4), and sel (1, 64, 1) picks the row whose full logits come back as logits_sel. Padding rows get a zero place column, so they write nothing.step is the same for one token and returns logits (1, 51866).place instead of a dynamic slice, which fell back to the CPU.Up to 224 tokens fit per take, prompt included.
Load decoder.mlmodelc three times with MLModelConfiguration.functionName set to cross, main and step and computeUnits = .cpuAndNeuralEngine. Make one state with main's makeState() and pass it to every call. Memory-map the two embedding files.
decoder.mlmodelc/weights/weight.bin 343 MBdecoder.mlmodelc/coremldata.bin 1.6 kBdecoder.mlmodelc/analytics/coremldata.bin 0.2 kBopenai/whisper-large-v3-turbo