Moonshine Streaming Tiny (English) converted to Core ML as five float16 models, built to transcribe speech on an Apple Watch while the user is still talking. The Swift package that runs them is swift-moonshine-coreml, and the code that converted them is moonshine-coreml-conversion.
This release, v1.1, was made from that repo's
v1.1 tag (commit
2dc4d21).
| File | Size | Runs on | What it does |
|---|---|---|---|
frontend.mlpackage |
4 MB | CPU only | 80 ms of audio → 4 feature frames |
encoder.mlpackage |
14 MB | CPU + Neural Engine | a 116-frame window of features → encoded frames |
adapter.mlpackage |
2.5 MB | CPU + Neural Engine | 4 encoded frames → 4 memory frames (adds position) |
cross_kv.mlpackage |
2.4 MB | CPU + Neural Engine | 4 memory frames → the decoder's cross-attention keys and values for them |
decoder_step.mlpackage |
41 MB | CPU + Neural Engine | one token in → scores for the next token; its self-attention cache is Core ML state |
tokenizer.bin |
Moonshine's tokenizer, token ids → text |
The models need watchOS 11, iOS 18 or macOS 15 (the decoder's cache uses MLState). On a watch they need a Series 9
or later: a Series 8's 2-core Neural Engine rejects them. Run the frontend on the CPU only
(computeUnits = .cpuOnly): on a Series 9's Neural Engine it computes wrong features.
Each model carries its release's tag as its Core ML version (metadata.versionString, for example v1.1), which stays
with it when Xcode compiles it into an app. Code can check that all five come from one release it was tested with, and
the Swift package does. v1.1 has the same weights as v1.0, with only the versions added; v1.0's models have none.
Download:
hf download dantrejo03/moonshine-streaming-tiny-coreml --local-dir moonshine-coreml
For an app, add the five .mlpackage folders to the Xcode target, and Xcode compiles each into a .mlmodelc (a watch
can't compile models itself). On a Mac or iPhone you can also compile at runtime. Loading the decoder and running one
step:
import CoreML
let compiledURL = try await MLModel.compileModel(at: URL(fileURLWithPath: "moonshine-coreml/decoder_step.mlpackage"))
let configuration = MLModelConfiguration()
configuration.computeUnits = .cpuAndNeuralEngine // .cpuOnly for the frontend
let decoderStep = try MLModel(contentsOf: compiledURL, configuration: configuration)
let state = decoderStep.makeState() // one per utterance
// inputs: an MLDictionaryFeatureProvider with token, position, k_cross, v_cross and cross_valid (shapes below)
let logits = try decoderStep.prediction(from: inputs, using: state).featureValue(for: "logits")
The whole pipeline (all five models, the state between calls and the tokenizer) is the Swift package
swift-moonshine-coreml. Its README says how to add it and how to
download a release it works with, and its tests check downloaded models against reference transcripts. It doesn't
include the models: add them to your app as above, and pass the folder Xcode compiles them into
(Bundle.main.resourceURL) to Moonshine.Transcriber(modelsDirectory:).
All float inputs and outputs are float32 except where noted; integers are int32.
| Model | Inputs | Outputs |
|---|---|---|
| frontend | audio_chunk [1, 1280] (16 kHz), conv1_buffer [1, 320, 4], conv2_buffer [1, 640, 4] |
features [1, 4, 320], conv1_buffer_out, conv2_buffer_out (pass these back in with the next chunk) |
| encoder | features [1, 116, 320], valid [1, 116] (1 for real frames, 0 for padding) |
encoded [1, 116, 320] |
| adapter | encoded [1, 4, 320], pos_offset [1] (memory frames so far) |
memory [1, 4, 320] |
| cross_kv | memory [1, 4, 320] |
k_new, v_new [6, 1, 8, 4, 40] (layer, batch, head, frame, head dim) |
| decoder_step | token [1, 1], position [1], k_cross, v_cross [6, 1, 8, 768, 64] float16, cross_valid [1, 768] |
logits [1, 1, 32768]; state k_self, v_self [6, 1, 8, 128, 64] float16 |
k_cross and v_cross hold up to 768 memory frames (15.36 s of audio). Each row's 40 numbers from cross_kv go in
its first 40 places, and the last 24 are padding the model never reads. Rows of 64 are what the Neural Engine wants:
with 40, Core ML re-lays out both inputs on every call. cross_valid is 1 for the frames written so far.
One recording (up to 15.36 s) is one utterance; everything starts over for the next.
cross_valid, and make one decoder state (makeState()) for the whole
utterance.k_cross/v_cross as float16.tokenizer.bin lists 32,768 tokens in id order, each as a length (1 byte under 128, otherwise 2: second × 128 + first
− 128) followed by its bytes. Join the bytes of every token, skipping special entries (over 2 bytes, starting with <
and ending with >). Turn ▁ into a space, trim spaces and tabs from both ends, then decode the bytes as UTF-8,
with each byte that doesn't begin a complete UTF-8 sequence becoming ? (as Moonshine's runtime does).
The Swift code that does all this is swift-moonshine-coreml.
Word error rate wasn't measured for this conversion. For the original model's (an average of 12.01% on the Open ASR Leaderboard), see its card.
decoder_step on the CPU and Neural Engine, not the CPU only: Core ML's CPU-only path writes its state wrongly.Only float16 is released, because the Neural Engine computes in float16. The conversion scripts also write float32
copies, used to measure how much accuracy float16 loses; rebuild them with the conversion repo's convert.py scripts.
MIT, like the original English model. The weights and tokenizer.bin are Moonshine AI's
(moonshine-ai/moonshine-v2); this release only converts them.
decoder_step.mlpackage/Data/com.apple.CoreML/weights/weight.bin 43 MBencoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin 15 MBfrontend.mlpackage/Data/com.apple.CoreML/weights/weight.bin 4 MBadapter.mlpackage/Data/com.apple.CoreML/weights/weight.bin 3 MBcross_kv.mlpackage/Data/com.apple.CoreML/weights/weight.bin 2 MBtokenizer.bin 250 kB