This 3-bit model is smaller but degraded on the 23-take, single-speaker Cantonese benchmark. With the same fixed prompt, normalized corpus CER worsened from 14.92% (4-bit) to 24.50% (3-bit). Without a prompt the gap was smaller, 16.37% to 18.26%, with insufficient evidence of equivalence. This is a shared experiment, not a drop-in quality-equivalent replacement. The 4-bit baseline remains recommended for the tested workflow.
These are affine 3-bit, group64 weights converted directly from the verified original FP32 Cantonese fine-tune, with FP16 conversion before quantization. Full Whisper large-v3, 32 encoder and 32 decoder layers, not Turbo. The weight file is 21.9% smaller than 4-bit, but primary-run MLX peak allocation was only about 10.8% lower; OS physical footprint savings were smaller and variable. No small-memory Mac was tested.
The separate 2-bit experiment collapsed to punctuation/whitespace and is not recommended. This repository contains 3-bit weights, never relabeled 2-bit or 4-bit weights.
Upstream: khleeloo/whisper-large-v3-cantonese, revision f48a890f78c7b6acf723f25d8c81e232ac7469ca, by khleeloo / Rita Frieske, fine-tuned from OpenAI Whisper large-v3 on Common Voice 17 Yue. Upstream model license: Apache-2.0. Conversion and evaluation by WinsonLi.
The retained original FP32 shards were hash-verified against the pinned Hub revision, cast to FP16, then directly affine-quantized. The existing packed 4-bit model was never used as input. All remapped parameter keys and shapes were checked, preserving MLX's generated alignment-head buffer. All 513 quantized matrices have the expected 3-bit packed width. The same conversion script reproduced the released 4-bit weights byte for byte, SHA256 7ea0e37f88ac3a4cdb00435a3418498ff707ba7b1cc45d1feb13bd82ec471374. Both 2-bit and 3-bit configurations load in the installed runtime and complete inference; load success is not accuracy success.
Original shards:
model-00001-of-00002.safetensors: 8c197eaef302259da30a98d352cc02e4914939d7c89b544430c59b2c35b06c07model-00002-of-00002.safetensors: d244e327e00cfaae4823fb9911bbe7605dc99b67cf650d15ca2b125f4750f8123-bit weights: 0dabaaea298485e7aeec4af7a99753264abc0392a257abcfda4afad93f714ac0, 685,896,454 bytes. Config and complete conversion provenance are included.
convert.py is the previously published sharded-Safetensors converter derived from Apple's mlx-examples commit 796f5b53cab69a3d48a44233ce21aae889e94a08, with strict key/shape checks and output-overwrite protection added. Its MIT license is included separately. The model remains Apache-2.0.
Use the package versions in provenance.json, obtain the authorized original checkpoint at its pinned revision, then:
python convert.py --torch-name-or-path /path/to/original-hf-checkpoint \
--mlx-path /path/to/new-3bit-model --dtype float16 \
--quantize --q-bits 3 --q-group-size 64
This uses the stock mlx-whisper loader's global quantization configuration. Research inference should explicitly use language yue, temperature 0, condition_on_previous_text=False, and fixed prompt conditions. See the method below; current InFlow dictionary/prompt/repair behavior must not be assumed equivalent to unprompted ASR.
| Variant | Weight MB | Raw corpus CER | Normalized corpus CER | Median decode s | RTF | MLX peak MiB | OS peak footprint MiB | Peak RSS MiB |
|---|---|---|---|---|---|---|---|---|
| 4-bit g64 | 877.696 | 29.04% | 14.92% | 0.661 | 0.0712 | 1672.3 | 2046.3 | 1098.0 |
| 3-bit g64 | 685.896 | 40.82% | 24.50% | 0.701 | 0.0771 | 1492.4 | 1897.9 | 918.3 |
| 2-bit g64 | 494.096 | 100.00% | 100.00% | 2.396 | 0.2580 | 1306.5 | 1595.2 | 734.1 |
| 2-bit g32 | 589.996 | 225.88% | 222.27% | 2.417 | 0.2598 | 1394.5 | 1682.3 | 821.1 |
The table uses the first prompted run for every variant, not the fastest repeat. MB is decimal bytes / 1,000,000; MiB is bytes / 1,048,576. Weight sizes are actual file lengths. Runtime files (weights plus config) occupy 877,703,168, 690,868,224, and 503,898,112 allocated bytes on this SSD for 4-, 3-, and 2-bit g64 respectively; filesystem allocation is not portable. The 2-bit g32 diagnostic weighs 589,996,337 bytes and is also unusable.
The 23 completed private Engineer Mode takes total 214.133 seconds and contain 20 distinct reading prompts in one Cantonese speaker's voice, with Cantonese and English technical speech. References are the prompts stored before recording, not newly human-verified verbatim transcriptions. There are repeated takes of two prompts, known number-vs-spoken-word mismatches, and one historically truncated recording. They are retained for comparison with the existing 4-bit evaluation. This is a small personal benchmark, not a population estimate or Common Voice test result. Recordings, references, transcripts, personal dictionaries, session IDs and private source passages are not distributed.
Every model used the same WAV bytes (hash-checked), mono 16 kHz PCM scaled by 1/32768, language yue, task transcribe, temperature 0, condition_on_previous_text=False, word_timestamps=False, fp16=True, sample length 224 per decoding window, and an explicit 64 MiB MLX cache limit. Prompted arms shared the same fixed InFlow mixed-Cantonese/English prompt. No personal dictionary, dynamic prompt context, hotwords, script correction, prompt-echo removal, silence gate, or LLM rewriting was applied. The prompt is kept with the private local experiment; no transcript or private vocabulary is being published. These are raw ASR measurements, not end-to-end dictation latency. Default temperature fallback was disabled in all arms.
Raw CER is Levenshtein edits / reference characters, including punctuation, whitespace and case. Normalized CER applies NFKC, Unicode casefold, removes whitespace and Unicode punctuation, and uses macOS CoreFoundation Traditional-Simplified on both reference and hypothesis, then totals edits / total reference characters (898 normalized reference characters; 1078 raw characters). It does not rewrite Cantonese into Mandarin, collapse number words into digits, forgive homophones, or repair names. CER can exceed 100% due to insertions. The 2-bit g32 score above 200% is intentional, not a percentage bug.
For continuity, the nonstandard per-take symmetric clean score (edits / max reference/hypothesis length, averaged) is 0.13242 for prompted 4-bit, 0.21384 for 3-bit, 1.00000 for 2-bit g64, and 0.96606 for 2-bit g32. This companion score is not conventional corpus CER. The historical 4-bit card's 0.132 value remains a legacy score, not a claim that the corrected CER is 13.2%. Historical term/marker substring scores are not reused; capitalization, script normalization, duplicate terms and substring matches made those scores unreliable.
With no prompt, normalized CER was 16.37% (4-bit), 18.26% (3-bit), and 100% (2-bit g64). Thus 3-bit was much closer to 4-bit without the static prompt. A paired bootstrap over 20 prompt clusters (5000 resamples; seed 20261112) estimates the 3-bit excess at +9.58 percentage points with the prompt (exploratory 95% interval +3.87 to +17.27), and +1.89 points without it (−0.24 to +4.08). The latter does not establish equivalence or a statistically clear degradation. No prompt was tuned after seeing results.
Prompted 3-bit had fewer normalized edits on 1 take, more on 15, and tied on 7. Both prompted 4-bit and prompted 3-bit produced byte-identical raw transcripts on a second fresh-process run. Their median decode times were 0.661/0.668 seconds and 0.701/0.705 seconds respectively. OS peak footprint varied from 1961.5–2046.3 MiB for 4-bit and 1846.9–1897.9 MiB for 3-bit. The table reports first runs consistently. The 2-bit no-prompt arm confirms the collapse is not dependent on the static prompt.
Apple M2 Max, 32 GiB unified memory; macOS 27.2.1, Python 3.13.7, MLX 0.32.2, mlx-whisper 0.4.3, NumPy 2.5.2. One model per fresh process, sequential runs, synchronized MLX work, one 1-second silent warm-up before measured samples. No source production preferences, worker code or installed model were modified. No GPU benchmark or converter ran concurrently; this was an ordinary desktop session, not a thermally controlled laboratory or an 8 GB Mac test.
MLX peak is mx.get_peak_memory() since reset before load, including loading, warm-up and decodes. It is an allocator statistic, not RSS. Process peak RSS and OS peak physical footprint come separately from macOS /usr/bin/time -l; current RSS/footprint samples use proc_pid_rusage. Sampled values are not labeled peaks. These overlapping memory measures must not be added together. The OS footprint numbers show real memory savings, but not a whole-machine RAM guarantee.
| Variant | Model load s | Silence warm-up s | Python package imports s |
|---|---|---|---|
| 4-bit | 0.341 | 2.834 | 1.138 |
| 3-bit | 0.084 | 2.900 | 1.187 |
| 2-bit | 0.105 | 2.726 | 0.924 |
These load times use the existing filesystem cache; they are not cold-boot measurements. RTF is summed decode seconds / summed audio seconds (lower is faster), excludes loading and warm-up, and is ASR-only. Peak memory includes loading and warm-up. Severe 2-bit output failure also took longer; fewer bits did not mean faster useful transcription.
Twelve pre-existing real-session recordings total 605.107 seconds: four read-aloud full turns, four noisy recordings, and four historical quiet comparators. They were selected from existing sets before seeing candidate outcomes. The read-aloud references cover only passages inside full turns with unscripted pre/postamble; the other eight have no verified human transcript. No CER is reported for these twelve. Disagreement between models is not ground-truth accuracy.
| Variant | Takes | Decode seconds | RTF | MLX peak MiB | OS peak footprint MiB | Empty after normalization |
|---|---|---|---|---|---|---|
| 4-bit | 12 | 34.200 | 0.0565 | 1727.4 | 1987.0 | 0 |
| 3-bit | 12 | 38.195 | 0.0631 | 1547.4 | 1810.3 | 1 |
| 2-bit g64 | 12 | 77.297 | 0.1277 | 1361.5 | 1628.5 | 12 |
The 3-bit/4-bit mean normalized transcript disagreement was 21.86% across these twelve recordings. This can identify material differences for later human listening; it cannot identify which model is right. Segment-level token-count flags are incomplete indicators: zero flags does not prove no decoding-window cap or truncation. All raw outputs and segment metadata are retained locally.
Keep 4-bit as the default for the tested prompted workflow. 3-bit can be investigated under a specific memory constraint, with human review of its prompt-sensitive errors; it is not established as equivalent. Uniform 3-bit group32 or a calibrated mixed-precision scheme are future hypotheses, not tested results here. MLX supports per-module quantization, but the stock mlx-whisper loader uses global quantization metadata; a mixed 3/4-bit model needs explicit module metadata and a compatible loader. No such modified model is included or claimed to work.
The private personal dataset cannot be independently reproduced from this repository alone. Reproducible conversion, evaluation code, methods and aggregate measurements are available in the public experiment repository. No private recordings, transcripts or dictionaries are published.
khleeloo/whisper-large-v3-cantonese