Granite Speech 5.0 470M TurboCTC — CoreML masked 4-bit

This is an unofficial Core ML conversion of IBM's Granite Speech 5.0 470M TurboCTC, with 4-bit linear weight quantization and a valid-length (mask) input, for on-device speech recognition on Apple Silicon Macs.

Source: ibm-granite/granite-speech-5.0-470m-turboctc at revision 91dcb0605daf65f88dbb04fef015c423241d941d. This is the multilingual release of 21 October 2026: English, French, German, Spanish, Portuguese, Italian and Japanese, with native punctuation and capitalization.

All model credit belongs to IBM Granite. This repository changes the format (a fixed-shape Core ML program that applies IBM's attention and padding masks from a valid_frames input), the weight precision (4-bit), one tokenizer setting, and adds change notices. IBM has not reviewed or endorsed it.

How this differs from the no-mask Core ML repositories

The earlier swaynehales/granite-speech-5.0-470m-turboctc-coreml-* repositories run every call as 16,384 zero-padded frames without a mask, and their no-mask fp16 package (like the no-mask reference computation) drops the final sentence period on 119 of 141 clips in our corpus. This repository adds the mask:

• Second input valid_frames (int32 [1]): the number of real feature frames before padding. The graph reproduces IBM's four masking steps (after the input projection, on attention keys, before each depthwise convolution, and halving the length after each subsampling layer).
• Second output valid_output_frames (int32 [1]) = floor(valid_frames / 4): read only that many frame_ids.
• Keeps the final sentence mark: it matches official inference on 138 of 141 clips (period or 。 at the end).
• Needs a loader that supports model_type granite_speech5_coreml_ctc_masked. The pinned Granite-MLX release (7ad6b19) does not load it yet: its loader accepts only granite_speech5_coreml_ctc and passes a single input. Until a loader supports it, use the Python example below.

Read this first: 4-bit costs accuracy

Prefer the masked 8-bit package (swaynehales/granite-speech-5.0-470m-turboctc-coreml-masked-8bit) unless size or memory matters more than accuracy. This 4-bit package is a disclosed size-over-accuracy option, with no accuracy gate claimed. Its cost is largest on Portuguese (6.59% against official inference) and on a private 5-minute recording (3.33%), and it adds a final period that official inference lacks on 3 of 141 clips.

Word or character difference in percent (normalized; lower is closer), measured through the Python path below:

FLEURS sets have only 10 clips each.

The original model

IBM describes the model as 16 conformer blocks trained with CTC, with a 16,384-token multilingual BPE head. It uses 8× temporal subsampling (100 Hz to 12.5 Hz), block attention over 128 frames, and self-conditioned CTC from the middle layer: 470M parameters in total. For training data, intended use, benchmarks and the technical report, see the IBM model card at the pinned revision.

Licence and permission to republish

IBM releases this model under the Apache License 2.0. The pinned source revision declares license: apache-2.0 in its card metadata, and its README states:

License: Apache 2.0

(source, revision 91dcb06)

Apache-2.0 section 2 grants the right to "reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form". Section 4 requires that redistribution include a copy of the licence and a notice of changes. Accordingly:

• LICENSE is the verbatim Apache-2.0 text from apache.org (SHA-256 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30). The source repository ships no separate LICENSE or NOTICE file at this revision.
• MODIFICATIONS.md states what was changed, and each changed file carries its own notice: the Core ML model's metadata has license: Apache-2.0 and a modification_notice field, and tokenizer_config.json has a _modification_notice key. Unchanged source files are byte-identical.

Do not confuse this model with ibm-granite/granite-speech-5.0-470m-turboctc-nc, a different, English-only model under CC-BY-NC-SA-4.0. That model was not used here.

What this conversion is

Required guard for consumers. Pass the true frame count, and check 1 <= T <= 16384 before calling (chunk longer audio). Outside that range the output count is not trustworthy: valid_frames 20000 returns valid_output_frames 5000 (more than the 4096 frames returned), and -1 returns -1. T of 1–3 returns 0 frames; treat it as an empty transcript. Passing a T larger than the real audio adds spurious trailing punctuation. Check that valid_output_frames == T // 4.

How it was produced. The graph is traced from a masked variant of Granite-MLX's explicit-ops PyTorch model (Granite-MLX commit 7ad6b19c698aac5aa8328c911e4a6eead456c5a1, classes reused unmodified, source state dict loaded strictly) that adds IBM's four masking steps, with padded attention keys filled with -1e4. It was converted with coremltools to fp16, keeping the length and mask computations in fp32 (coremltools otherwise lowers the integer length halving to a float16 operation that rounds lengths above 2048). The weights were then compressed with Granite-MLX quantize_granite_coreml.py --method linear --bits 4 --granularity per-block --block-size 32 --weight-threshold 2048, run unmodified; the internal masks were checked exact against the expected prefix masks at 22 lengths after compression. Versions: coremltools 9.0 with torch 2.7.0 for conversion; reference inference with transformers 5.16.1 and torch 2.13.0. Run on macOS 27.0.1, Apple M2 Max, on 2026-11-13/14 (America/Los_Angeles). Full provenance is in CONVERSION.json.

Files

Every file's SHA-256 is listed in SHA256SUMS; check with shasum -a 256 -c SHA256SUMS.

Tokenizer deviation. tokenizer_config.json adds "clean_up_tokenization_spaces": false and a _modification_notice string (the Apache-2.0 §4(b) change notice; Python transformers and swift-transformers both load the file with it, tested); nothing else differs from the source file (source SHA-256 2f61ece3468bf7ca7a1b6dd3fd5b809814315bea9c298feea035fda4931b9f02). swift-transformers 1.3.3 turns that cleanup on when the key is absent, and it strips the space before ? ! : ;, which corrupts French typography. Python transformers already leaves it off for this tokenizer, so Python output is unchanged.

How to use it

Python: coremltools

Tested with coremltools 9.0, transformers 5.16.1 and soundfile on macOS 27, on a local copy of this repository and 16 kHz mono audio of at most 327.68 s (this exact script was run on this folder):

import numpy as np, soundfile as sf, coremltools as ct
from transformers import AutoProcessor

repo = "/path/to/granite-speech-5.0-470m-turboctc-coreml-masked-4bit"
audio, sr = sf.read("audio.wav", dtype="float32")             # 16 kHz mono
proc = AutoProcessor.from_pretrained(repo)
feats = proc(audio, sampling_rate=16000, return_tensors="np")["input_features"][0]   # [T, 320]
T = len(feats)
assert 1 <= T <= 16384, "valid_frames must be 1..16384; chunk longer audio"          # required range guard
x = np.zeros((1, 16384, 320), np.float16); x[0, :T] = feats                          # fixed shape, zero-padded
model = ct.models.MLModel(f"{repo}/GraniteSpeech.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_GPU)
out = model.predict({"features": x, "valid_frames": np.array([T], dtype=np.int32)})
n = int(out["valid_output_frames"].reshape(-1)[0])
assert n == T // 4                                                                   # returned-count guard
ids = out["frame_ids"].reshape(-1)[:n]                       # frames beyond n are undefined
collapsed, prev = [], None
for t in ids.tolist():                      # greedy CTC: merge repeats, drop blank (0)
    if t != prev and t != 0: collapsed.append(t)
    prev = t
text = proc.tokenizer.decode(collapsed, group_tokens=False).replace("<|nospeech|>", "").strip()
print(text)

Pass group_tokens=False when decoding IDs that are already collapsed; otherwise the tokenizer merges repeated tokens a second time. For longer audio, chunk it as Granite-MLX does: 286.72 s core windows with up to 20.48 s of context on each side; run each segment with its own T; trim the context frames from each segment's frame_ids in frame space (12.5 output frames per second); concatenate the retained frame IDs of all segments; then CTC-collapse once and decode.

Swift / Granite-MLX

Not yet supported by the pinned Granite-MLX release (7ad6b19); see "How this differs" above.

Validation

Method. The reference is IBM's model in transformers 5.16.1, fp32 on CPU, variable length with an attention mask, with greedy CTC (official inference). This package was run through the Python path above (Core ML, CPU and GPU), with zero-padded features and valid_frames = T.

The fixed corpus has 141 unique clips:

• 73 from LibriSpeech (hf-internal-testing/librispeech_asr_dummy);
• 60 from the FLEURS test split (the first 10 rows in file order for each of fr, de, es, pt, it and ja);
• 4 English local clips: the public-domain JFK inaugural excerpt (11 s) and 3 private recordings (30 s, 1 min, 5 min) that are not published;
• 4 non-speech clips: digital silence of 5 s and 30 s, and seeded white and pink noise.

Long-audio behaviour was tested separately on 4 constructed clips of 286.5–328 s. Text is compared after NFKC normalization, lower-casing and removing punctuation; WER is used for European languages, and CER with whitespace removed for Japanese. All rates are corpus-level per set. With 10 clips per FLEURS language these are small samples: they show conversion fidelity, not benchmark accuracy.

Results for this 4-bit package, in percent:

Final sentence mark: matches official inference on 138/141 clips (on 3 clips it adds a period that official inference lacks; it drops one on none of the 141 tested clips). Local recordings (4 clips): 3.15% against official inference.

Long audio. Using the masked package per segment with Granite-MLX's 286.72 s chunking, compared with chunked official inference: 1.24–1.43% on the four constructed boundary clips and 3.33% on a private 5-minute recording.

Size and speed. The weight file is 282,791,808 bytes. Python predict median over 5 rounds after a warm-up: 0.50 s for an 11 s clip and 0.50 s for a full 327.5 s window (the graph always runs all 16,384 frames), process peak RSS 1.17 GB. These timings were measured on a busy machine (1-minute load average about 6) and are indicative only.

Limitations

• Close to IBM's own inference, not identical. Two sources, measured separately: fp16 arithmetic, measured on the masked fp16 package, changes two one-token near-ties (below); 4-bit quantization adds further differences, measured on this package against masked fp16 (first column of the table; worst fleurs-pt at 6.59%). Against official inference the worst set is fleurs-pt at 6.59%.
• Original acceptance gates for the masked fp16 package not met, and disclosed. The masked fp16 package (from which this one is compressed) matched official inference by collapsed tokens on 139 of 141 clips; the original gate was 141 of 141 and is not met. The two differences are one-token fp16 near-ties (logit margins 0.0034 and 0.0020). On long audio, the original gate (exact 5 of 5 against the chunked official reference) is not met at 4 of 5: one lost period at the 287.0 s boundary clip, 0% normalized word difference. Both were accepted as disclosed limits.
• 4-bit cost. No accuracy gate is claimed; it was accepted as a disclosed size-over-accuracy option on the numbers above (worst fleurs-pt 6.59% against official inference; 3.33% on 5-minute audio; 3 extra final periods).
• Silence and noise. This package reproduces official inference on digital silence and noise: 5 s of digital silence returns a spurious "Il.", as official inference does; 30 s of silence returns "."; white and pink noise return the literal <|nospeech|> token. Filter <|nospeech|> and punctuation-only output in your application.
• Out-of-range valid_frames gives untrustworthy counts (see the required guard above).
• No Swift loader yet. Granite-MLX 7ad6b19 rejects granite_speech5_coreml_ctc_masked; the Swift path is untested.
• Granite-MLX does not verify the weight hash (tested on the no-mask packages). Verify SHA256SUMS yourself.
• Native punctuation. The model punctuates and capitalizes by itself. If your application also runs a separate punctuation or formatting model, expect overlap.
• No Neural Engine placement, quiet-host latency, streaming or iOS measurements were made.