Moonshine Tiny — QAT for int8 deployment

Quantisation-aware-trained checkpoint of UsefulSensors/moonshine-tiny, prepared for int8 inference on a RISC-V SoC (Rocket + a custom RoCC matrix engine on a Xilinx Zynq-7020).

The weights in this repository are float32. They are trained so that they quantise cleanly to int8 — they are not stored as int8. Use them as the starting point for int8 quantisation, not as a pre-quantised model.

Why this exists, and why the stock checkpoint will not do

Post-training quantisation of the stock checkpoint does not survive this deployment. Measured on LibriSpeech dev-clean (765 utterances), float simulation, at three activation-calibration policies:

At the deployed setting the stock checkpoint has roughly double the word error rate. A pipeline that silently fetches the stock model still runs, still produces plausible transcripts, and is twice as wrong — with nothing in the output to say why. That failure mode is the reason this checkpoint is published rather than kept locally.

Deployment settings that matter

Reproducing the numbers above requires the calibration policy as well as the weights.

• Activation ranges: p99.9, pooled over the whole calibration set — not the per-sample percentile then max, which gives ranges 1.09× (median) to 3.11× (worst layer) wider and costs about 5 points.
• Calibration set: 64 pinned LibriSpeech dev-clean windows.
• Not p99.99, even though that is what QAT trained against. p99.9 over the deployment calibration windows lands about 5.1 points better. The mismatch is deliberate and measured.

Measured results

The last row is end-to-end from a microphone on the device — acoustic path included.

Provenance

Files

model.safetensors          108,389,160 B
config.json                        864 B
generation_config.json             189 B
preprocessor_config.json           215 B
tokenizer.json                 1,985,530 B

Licence

MIT, following the base model. Check the base model's terms if you redistribute.