Ear — 16MB on-device speech-to-text

Ear is a tiny English speech-to-text model designed for on-device inference. It prioritizes extreme smallness over raw accuracy: int8 weights, no tokenizer or vocab file, host-side mel frontend, greedy CTC decoder, published failure modes, and honest benchmarks against Whisper.

---

Quick start

import numpy as np
import soundfile as sf
import onnxruntime as ort
import json, struct

# Load model artifacts
meta   = json.load(open("artifacts/stt_meta.json"))
labels = json.load(open("artifacts/labels.json"))
ALPHABET = [e["symbol"] for e in labels]

sess = ort.InferenceSession("artifacts/stt.onnx",
                             providers=["CPUExecutionProvider"])

# Load torchaudio's exact filterbank (extract once and cache as .npy)
# See notebook Section 06 for the one-time extraction step.
MEL_FB = np.load("artifacts/mel_filterbank.npy")          # (201, 80)
WIN    = (0.5 * (1.0 - np.cos(2.0 * np.pi * np.arange(400) / 400))).astype(np.float32)

def log_mel(wav: np.ndarray) -> np.ndarray:
    """wav: float32 mono @16kHz → (80, T) log-mel."""
    wav = np.pad(wav, 200, mode="reflect")
    n_frames = 1 + max(0, (len(wav) - 400) // 160)
    mel = np.empty((80, n_frames), dtype=np.float32)
    for i in range(n_frames):
        seg = wav[i*160:i*160+400] * WIN
        spec = np.abs(np.fft.rfft(seg, n=400)) ** 2
        mel[:, i] = MEL_FB.T @ spec
    return 10.0 * np.log10(np.maximum(mel, 1e-10))

def greedy_ctc(log_probs, blank=0, thresh=0.35):
    probs = np.exp(log_probs)
    ids, conf = probs.argmax(-1), probs.max(-1)
    out, prev, cs = [], -1, []
    for t, i in enumerate(ids):
        if i != blank:
            cs.append(float(conf[t]))
            if i != prev: out.append(ALPHABET[i])
        prev = int(i)
    text = "".join(out)
    c = sum(cs)/len(cs) if cs else 0.0
    return (text, c) if c >= thresh else ("[uncertain]", c)

def transcribe(path: str):
    wav, sr = sf.read(path, dtype="float32")
    assert sr == 16000, "resample to 16kHz first"
    if wav.ndim > 1: wav = wav.mean(1)
    mel = log_mel(wav).astype(np.float32)
    m = mel.T[None, None]                          # (1,1,T,80)
    logits, out_len = sess.run(None, {
        "mel": m, "mel_len": np.array([mel.shape[1]])
    })
    T = int(out_len[0])
    return greedy_ctc(logits[0, :T])

text, confidence = transcribe("audio.wav")
print(f"[{confidence:.2f}] {text}")

---

Artifacts

int8 bin format

Any host can load stt_int8.bin in ~30 lines:

import struct, json, numpy as np

def load_int8_bin(path):
    raw = open(path, "rb").read()
    (hl,) = struct.unpack("<I", raw[:4])
    header = json.loads(raw[4:4+hl])
    blob   = raw[4+hl:]
    weights = {}
    for name, t in header["tensors"].items():
        b = blob[t["offset"]:]
        if t["dtype"] == "int8":
            n = int(np.prod(t["shape"]))
            q = np.frombuffer(b[:n], dtype=np.int8).reshape(t["shape"])
            weights[name] = q.astype(np.float32) * t["scale"]
        else:
            n = int(np.prod(t["shape"]))
            weights[name] = np.frombuffer(b[:n*4], dtype=np.float32).reshape(t["shape"]).copy()
    return weights

---

Architecture

16kHz waveform (host)
  → 80-dim log-mel, 25ms window / 10ms hop (host, fixed constants)
  → Conv2d 4× subsampling → 40ms frames
  → 9 × Pre-LN ConformerBlock (d=256, 4 heads, conv kernel 15, FFN 4×)
  → Linear → 29 CTC classes
  → (host) greedy CTC decode → text + confidence

Key design choices:

• Manual QKV Linear layers (not nn.MultiheadAttention) — every large matmul is int8-quantizable.
• BatchNorm folded into depthwise convs before export — BN is never in the shipped artifact.
• Greedy CTC decoder fits in 20 lines of numpy — no inference runtime needed.
• Character-level alphabet, hardcoded — zero BPE files, zero vocab files.

Model definition: model_lib.py in this repo.

---

Training

Training notebook: tongue_stt_v3-FINAL.ipynb. Checkpoints saved every 2000 steps to Google Drive; training survived a 24h session restart via resume.

---

Results

Gate scorecard

WER on LibriSpeech test-clean

*Whisper measured on 200-utterance ≤3s subset; Ear WER on same subset is 13.4%.

Ear is 9.4× smaller than Whisper tiny.en and 1.6× worse WER on the full test-clean. The win is size × latency, not raw accuracy.

---

Failure modes

These are published as part of the product — honesty about what the model can't do is as important as the benchmarks.

1. Proper nouns and names — structural No language model means no lexical anchoring. Irish, Greek, and archaic English names from LibriSpeech's literary corpus (Joyce, Plato, Shakespeare) fail badly.

• "stephanos dedalos" → "stuffronors daelos"
• "timaeus approves" → "to me as a proves"

2. Short utterances (≤3s) — harder by design 12.5% WER vs 8.6% full-set (3.9pp gap). Single phrases give less temporal context for error recovery.

3. Apostrophe handling Contractions occasionally produce spurious apostrophe placement.

• "he's swiftly punished" → "he 'is swiftly punished"

4. English clean speech only Trained on LibriSpeech clean read speech. Accents, noise, and far-field audio degrade sharply. Out of scope by design.

5. No language model in the shipped artifact Greedy CTC on characters — homophones and phonetically similar words are indistinguishable. Reported honestly, not hidden.

6. Confidence threshold is uncalibrated uncertain=0.0% across all 2,620 test utterances. The confidence signal exists (mean non-blank CTC posterior) but has not been calibrated against accuracy. Do not rely on it for gating.

---

License

Code: Apache-2.0. Weights trained on LibriSpeech (CC-BY-4.0) — trained weights inherit CC-BY-4.0.