Granite Speech 5.0 470M TurboCTC — CoreML 4-bit

This is an unofficial Core ML conversion of IBM's Granite Speech 5.0 470M TurboCTC, with 4-bit linear weight quantization, for on-device speech recognition on Apple Silicon Macs.

Source: ibm-granite/granite-speech-5.0-470m-turboctc at revision 91dcb0605daf65f88dbb04fef015c423241d941d. This is the multilingual release of 21 October 2026: English, French, German, Spanish, Portuguese, Italian and Japanese, with native punctuation and capitalization.

All model credit belongs to IBM Granite. This repository changes the format (a fixed-shape Core ML program with no attention mask, which behaves slightly differently from IBM's own inference; see Limitations), the weight precision (4-bit), one tokenizer setting, and adds change notices. IBM has not reviewed or endorsed it.

Read this first: 4-bit costs accuracy

Prefer the 8-bit package (swaynehales/granite-speech-5.0-470m-turboctc-coreml-8bit) unless size or memory matters more than accuracy. This 4-bit package is a disclosed size-over-accuracy option, with no accuracy gate claimed. Its cost is several times the 8-bit cost, and it is largest on Portuguese (7.31% against fp16). It is no faster than 8-bit in our timing.

Word or character difference in percent (normalized; lower is closer), measured through the Granite-MLX Swift CLI:

FLEURS sets have only 10 clips each.

The original model

IBM describes the model as 16 conformer blocks trained with CTC, with a 16,384-token multilingual BPE head. It uses 8× temporal subsampling (100 Hz to 12.5 Hz), block attention over 128 frames, and self-conditioned CTC from the middle layer: 470M parameters in total. For training data, intended use, benchmarks and the technical report, see the IBM model card at the pinned revision.

Licence and permission to republish

IBM releases this model under the Apache License 2.0. The pinned source revision declares license: apache-2.0 in its card metadata, and its README states:

License: Apache 2.0

(source, revision 91dcb06)

Apache-2.0 section 2 grants the right to "reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form". Section 4 requires that redistribution include a copy of the licence and a notice of changes. Accordingly:

• LICENSE is the verbatim Apache-2.0 text from apache.org (SHA-256 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30). The source repository ships no separate LICENSE or NOTICE file at this revision.
• MODIFICATIONS.md states what was changed, and each changed file carries its own notice: the Core ML model's metadata has license: Apache-2.0 and a modification_notice field, and tokenizer_config.json has a _modification_notice key. Unchanged source files are byte-identical.

Do not confuse this model with ibm-granite/granite-speech-5.0-470m-turboctc-nc, a different, English-only model under CC-BY-NC-SA-4.0. That model was not used here.

What this conversion is

How it was produced. The converter scripts were run unmodified from Granite-MLX at commit 7ad6b19c698aac5aa8328c911e4a6eead456c5a1:

• convert_granite_coreml.py --feature-frames 16384 --output-mode argmax --minimum-target macos15 built the fp16 program.
• quantize_granite_coreml.py --method linear --bits 4 --granularity per-block --block-size 32 --weight-threshold 2048 then quantized it.

Versions: coremltools 9.0 with torch 2.7.0 for conversion; reference inference with transformers 5.16.1 and torch 2.13.0. Run on macOS 27.0.1, Apple M2 Max, on 2026-11-13 (America/Los_Angeles). Full provenance is in CONVERSION.json.

Rejected alternative. A 4-bit k-means palettization (per-grouped-channel, group size 16) was also built and rejected. Against fp16 it differed by LibriSpeech en 10.45, fr 18.33, de 15.98, es 15.45, pt 21.15, it 10.64, ja 15.23 percent (Python path).

Files

Every file's SHA-256 is listed in SHA256SUMS; check with shasum -a 256 -c SHA256SUMS.

Tokenizer deviation. tokenizer_config.json adds "clean_up_tokenization_spaces": false and a _modification_notice string (the Apache-2.0 §4(b) change notice; Python transformers and swift-transformers both load the file with it, tested); nothing else differs from the source file (source SHA-256 2f61ece3468bf7ca7a1b6dd3fd5b809814315bea9c298feea035fda4931b9f02). swift-transformers 1.3.3 turns that cleanup on when the key is absent, and it strips the space before ? ! : ;, which corrupts French typography. Python transformers already leaves it off for this tokenizer, so Python output is unchanged.

How to use it

Swift / command line: Granite-MLX

Tested with the granite-mlx CLI 0.1.1, built from Granite-MLX 7ad6b19 with swift build -c release --product granite-mlx (Swift 6.4), on a local copy of this repository:

granite-mlx transcribe audio.wav --model /path/to/granite-speech-5.0-470m-turboctc-coreml-4bit --backend coreml \
  --no-punctuate --output-format txt --output-dir out

--no-punctuate disables Granite-MLX's separate punctuation model; this Granite release punctuates natively. Use --output-format txt: compatibility was validated for raw text only. The CLI's default subtitle output (SRT) and other word-timed formats run words together with this tokenizer (see Limitations). Loading by Hugging Face repository id was not tested.

Python: coremltools

Tested with coremltools 9.0, transformers 5.16.1 and soundfile on macOS 27, using a local copy of this repository and 16 kHz mono audio of at most 327.68 s:

import numpy as np, soundfile as sf, coremltools as ct
from transformers import AutoProcessor

repo = "/path/to/granite-speech-5.0-470m-turboctc-coreml-4bit"
audio, sr = sf.read("audio.wav", dtype="float32")            # 16 kHz mono
proc = AutoProcessor.from_pretrained(repo)
feats = proc(audio, sampling_rate=16000, return_tensors="np")["input_features"][0]   # [T, 320]
x = np.zeros((1, 16384, 320), np.float16); x[0, :len(feats)] = feats                 # fixed shape, zero-padded
model = ct.models.MLModel(f"{repo}/GraniteSpeech.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_GPU)
ids = model.predict({"features": x})["frame_ids"].reshape(-1)[: len(feats) // 4]      # valid frames = floor(T/4)
collapsed, prev = [], None
for t in ids.tolist():                      # greedy CTC: merge repeats, drop blank (0)
    if t != prev and t != 0: collapsed.append(t)
    prev = t
print(proc.tokenizer.decode(collapsed, group_tokens=False))

Pass group_tokens=False when decoding IDs that are already collapsed. Otherwise the tokenizer merges repeated tokens a second time.

Validation

Method. The reference is IBM's model in transformers 5.16.1, fp32 on CPU, with greedy CTC (official variable-length inference, and a no-mask fixed-shape reference for the Core ML contract). This package and our fp16 and 8-bit packages were run through the Granite-MLX Swift CLI on the same corpus.

The fixed corpus has 141 unique clips:

• 73 from LibriSpeech (hf-internal-testing/librispeech_asr_dummy);
• 60 from the FLEURS test split (the first 10 rows in file order for each of fr, de, es, pt, it and ja);
• 4 English local clips: the public-domain JFK inaugural excerpt (11 s) and 3 private recordings (30 s, 1 min, 5 min) that are not published;
• 4 non-speech clips: digital silence of 5 s and 30 s, and seeded white and pink noise.

Long-audio behaviour was tested separately on 4 constructed clips of 286.5–328 s. Text is compared after NFKC normalization, lower-casing and removing punctuation; WER is used for European languages, and CER with whitespace removed for Japanese. All rates are corpus-level per set. With 10 clips per FLEURS language these are small samples: they show conversion fidelity, not benchmark accuracy.

Further results for this 4-bit package, in percent (the cost table is at the top):

Size and speed. Weight file 282,676,928 bytes (8-bit: 691,520,192). Measured with the Granite-MLX CLI, 5 rounds interleaved with the 8-bit package: median 0.769 s on an 11 s clip and 1.252 s on a 328 s clip (8-bit: 0.767 s and 1.234 s); peak RSS 0.62 GB (8-bit 1.44 GB). These timings were measured on a busy machine (1-minute load average about 6–8) and are indicative only.

Limitations

• 4-bit cost. See the table at the top. This package is noticeably less accurate than the 8-bit and fp16 packages, most of all on Portuguese, French and Japanese.
• Fixed shape and no mask: not identical to IBM's own inference. The package always runs 16,384 frames, with zero padding and no attention mask. Measured on the no-mask fp16 reference over the 141 clips, the final sentence mark (. or 。) is dropped on 119 clips compared with official variable-length inference. This 4-bit package itself, through the Swift CLI, dropped it on 110 of the 141 clips. Its other 4-bit differences come on top of that.
• On silence this package returns <|nospeech|> or an empty string, like the fp16 package (all 4 non-speech clips match the no-mask reference).
• Granite-MLX does not verify the weight hash (tested on the fp16 and 8-bit packages). Verify SHA256SUMS yourself.
• Silence and noise. The model emits a literal <|nospeech|> token, which decode returns as text unless you filter it.
• Long audio. Granite-MLX chunks audio longer than 286.72 s (when run with --audio-chunk-duration 286.72 --audio-chunk-context 20.48, as tested) into windows with 20.48 s of context on each side, trimming context in frame space before CTC collapse. Results near chunk joins depend on that policy (see Validation).
• Swift tokenizer. Granite-MLX 7ad6b19 loads this tokenizer only by forcing the GPT-2 tokenizer class (the source class is ParakeetTokenizer, which swift-transformers 1.3.3 rejects). Decoding is correct for all seven languages in our tests when the shipped tokenizer_config.json is used.
• Subtitle and word-timing output in Granite-MLX 7ad6b19. The CLI's default SRT output (and other word-timed formats) joins words without spaces, for example Andso,myfellowAmericans,..., because the timing path decodes one token at a time and this tokenizer's decoder strips each token's leading space. Raw text (--output-format txt) is correct. A fix belongs in Granite-MLX.
• Native punctuation. The model punctuates and capitalizes by itself. If your application also runs a separate punctuation or formatting model, expect overlap.
• No Neural Engine placement, quiet-host latency, streaming or iOS measurements were made.