Whisper tiny.en, hand-edited to stop writing words on silence

An unofficial, hand-edited version of openai/whisper-tiny.en. It wasn't retrained: 3,845 of its 37.8 million weights were changed by hand.

On audio with nobody speaking, such as silence, hiss, hum, tones or noise, the original writes words. With greedy decoding it's almost always "you", and with the pipeline's beam search it's often looping sentences like "I'm going to do it. I'm going to do it." This version writes a lone "." instead. Its transcripts of real speech are as accurate as before.

Made with WSP Lab, coming soon: join the waitlist at wsp-lab.ai

WSP Lab is a tool that lets AI agents edit a model's weights directly, quickly and verifiably. This edit used its approach, with a purpose-built harness.

How long it took

• Total: 2 hours 17 minutes from the start of the run to the upload, on one laptop (an Apple M5 Max), with no training. It was 2 hours 19 minutes to the download check below.
• Download check: downloaded with no login into an empty cache, it wrote no words on 400 fresh non-speech clips, with either decoding. Its greedy transcripts of those clips and of 100 utterances matched the experiment's exactly.
• Budget: the edit used 2 test runs of about 20 seconds each. Most of the 40 minutes went to the agent's reasoning, not to compute.
• Timing source: every time above comes from the experiment's own logs and file timestamps, on 6 and 7 November 2026.

Results

All numbers are on held-out data that the editors never saw.

• Non-speech: seeded synthetic 10-second clips, of 8 kinds (digital silence, hiss, white noise, rumble, mains hum, tones, chords and clicks).
• Speech: 820 LibriSpeech test-clean utterances (items 1,800 to 2,619, sorted by id), scored with Whisper's English text normaliser.

A clip counts as a hallucination if its transcript contains any letter or digit.

Paired differences on the same utterances, with 95% bootstrap intervals:

• Greedy: WER +0.000 points (−0.037 to +0.032). 10 of the 820 transcripts changed: 2 got better and 3 worse.
• 5-beam search: WER −0.031 points (−0.068 to −0.006).

Sounds nobody tested during editing. There were 592 clips:

• 200 instrumental music loops;
• the 14 macOS alert sounds, each at 3 positions in silence;
• 350 clips of 7 new noise types (pink, brown, chirps, square waves, band-passed, modulated and speech-shaped).

The last three columns are this model's rate on each kind of sound.

Harder speech, with greedy decoding: under white noise at 10 dB SNR, quieted by 26 dB, after 8 seconds of silence, and after 5 seconds of hiss, every WER change was within ±0.08 points. 2 of the 820 quiet utterances came out as "." (see Limits).

Timestamp mode (return_timestamps=True): words on non-speech fell from 49.1% to 0%, and WER changed by −0.037 points.

Long recordings, tested after the release. There were 273 recordings of about 2.5 minutes, each with 3 LibriSpeech utterances and stretches of 5 to 60 seconds of generated noise or silence around them, and 50 recordings of 90 seconds of non-speech only.

Each cell is the original, then this model. It missed no more of the spoken words in any of the three. For long audio, use chunk_length_s=30:

• sequential decoding through the pipeline loops badly with either model;
• with greedy sequential generate(), this edit doesn't stop hallucinations in recordings that also contain speech.

Compared with a rule in the decoder

The original already computes the signal this edit uses: its probability of "no captions" (token 50361) at the first decoding position. OpenAI's own transcribe() reads it to skip silent segments. So, after the release, the original was also scored with two rules in the decoder, on the same held-out data:

• If you can change your decoding code, the plain threshold does about as well, and it outputs an empty string rather than ".". It skipped no real speech in any test, where this model blanked 4 short or quiet clips.
• OpenAI's default rule does much worse. It also needs the transcript to look unconfident, and these hallucinations often aren't: it keeps " you" on every clip of digital silence.
• transformers 5.22.2's built-in version, generate(no_speech_threshold=0.6, logprob_threshold=-1.0), fails with greedy decoding, and with beams it skipped none of 16 test clips.
• What this model adds is that it needs no code: it works in pipeline() and generate() as they are.

What was changed

The original already knows when nobody is speaking. At the first decoding position, its own "no captions" prediction separates speech from non-speech perfectly, but the first word is chosen one position later, and that prediction isn't used there. The edit carries the signal across, in three steps, each built to stay exactly zero on speech:

1. Detect. Two unused units in decoder layer 2's MLP (267 and 718) form a soft step. It reads the model's own "no captions" direction at the first position and fires above a threshold. When it fires, it writes a marker direction, borrowed from a decoder position embedding (447) that decoding never reaches.
2. Carry. One value channel (dimension 145) of attention head 2 in decoder layer 3 carries the marker forward. That head already looks mostly at the first position.
3. Act. Two unused units in decoder layer 3's MLP (1426 and 621) form a second step. When the marker arrives, they raise " ." and end-of-text and lower " you" and " You".

Whisper's decoding config won't let the model end at its first token, so on non-speech it writes " ." and then ends. Every direction used comes from the model's own weights, and every number was chosen by hand. On 1,620 held-out utterances the detector and the final step were exactly zero at all 41,984 decoding positions.

edit.safetensors holds just the 9 changed tensors. model.safetensors is the complete model.

Usage

import numpy as np
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="erickvs/whisper-tiny.en-hand-edited")

print(asr("speech.wav")["text"])  # transcribes as usual
print(asr("long_recording.wav", chunk_length_s=30)["text"])  # for audio over 30 seconds

noise = (0.05 * np.random.default_rng(0).standard_normal(16000 * 10)).astype(np.float32)  # 10 s of white noise
print(asr({"raw": noise, "sampling_rate": 16000})["text"])  # " ."; the original writes " I'm going to do it. I'm going to do it. ..."

To apply just the edit to the original weights:

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")
edit = load_file(hf_hub_download("erickvs/whisper-tiny.en-hand-edited", "edit.safetensors"))
model.load_state_dict(edit, strict=False)  # replaces 9 tensors and leaves the rest untouched

It was tested with transformers 5.22.2 and torch 2.15.0, on Apple's MPS backend, in float32.

Limits

• It writes "." rather than nothing. Whisper's generation config blocks end-of-text as the first token, so the shortest possible transcript is one token. Strip a lone "." if you need an empty string.
• Mostly synthetic non-speech. The clips it was tested on were generated noises plus some music loops and alert sounds. A third of the alert sounds still get words. Real recordings, such as traffic, crowds, rooms or music with singing, haven't been tested.
• Rare false alarms. The decision is made at the start of each 30-second window, where its margin on speech is small: 12.76 at the most on held-out speech, against a threshold of 12.93. It turned 2 of 820 test utterances quieted by 26 dB into ".". Both were one- or two-word replies, "Indeed." and "Quick, quick.". It did the same to 2 of 820 three-second fragments of speech.
• Speech coverage: read English from LibriSpeech only. Other accents, conversational speech and noisy real-world recordings haven't been tested. Long recordings were tested only with generated noise between utterances (see Results).
• One base model: the numbers were chosen for whisper-tiny.en, and they won't carry over to other Whisper sizes.
• Comparison: Calm-Whisper (2025) cut such hallucinations by more than 80% by fine-tuning. This edit wasn't compared with it on the same data.

How it was made

The model comes from an experiment in editing neural networks by hand, with no training. Three AI editors (Claude Fable 5.1, by Anthropic) worked independently under the same fixed rules. Those banned gradients, optimizers, fitting weights to data, automated search and weights from other models. Each editor had 40 test runs on development data. A separate selection split was used to choose between the finished edits, and a final split, which no editor could run, gave one final score. The bar was set before any edit existed: cut non-speech hallucinations by at least 80%, with LibriSpeech word error no worse by a paired bootstrap interval.

This isn't the edit the pre-registered rule picked. The rule picked a different editor's edit. It was even cleaner with greedy decoding (no held-out transcript changed), but under pipeline()'s default 5-beam search it made hallucinations worse than the original's: 97% against 23%. The audit found this after the final scores were in. This edit also met the bar on the final split, and it holds under both kinds of decoding, so it was released instead. The switch rests on the selection split, not the final one. There, under 5-beam search, the pick wrote words on 97.2% of non-speech clips, where the original wrote them on 22.8%. This edit wrote them on none, with a WER change of +0.025 points (−0.013 to +0.071). The third edit failed the speech condition there.

An independent audit recomputed the headline numbers from the raw transcripts, checked that the edit file holds only what its author described, and tested its explanation on held-out data. The full write-up, including the other two edits and everything that went wrong, is in the experiment notes.

Support the experiments

These experiments run on one laptop, but the AI agents that make and audit the edits cost real money: the three editors behind this release used about 830,000 tokens. If you'd like more models fixed by hand like this one, published with their full results and limits, you can support the experiments on Patreon. It pays for the tokens behind the next ones.

Credits and license

Derived from openai/whisper-tiny.en by OpenAI (Robust Speech Recognition via Large-Scale Weak Supervision), and released under the same Apache License 2.0. The license text is in LICENSE, and NOTICE lists the modifications. LibriSpeech (CC BY 4.0) was used only for evaluation.

This model isn't affiliated with or endorsed by OpenAI.