A separate draft model for greedy speculative decoding with Whisper large-v3 (openai/whisper-large-v3) in
English, Chinese, Japanese and Cantonese. It is distil-whisper/distil-large-v3 fine-tuned on large-v3's own
transcripts. distil-large-v3 was trained on English only and accepts almost nothing on other languages as a draft;
this model accepts 5.2 to 5.7 tokens per round on all four.
| setting | value |
|---|---|
| architecture | WhisperForConditionalGeneration, distil-large-v3's: 32 encoder layers, 2 decoder layers, hidden 1280 |
| encoder | bit-identical to large-v3's in bf16 (checked on all 487 tensors), so a runtime can pass the target's encoder output |
| trained | the decoder only, with its tied embedding: 119.4M parameters |
| vocabulary | large-v3's, 51,866 |
| precision | bf16 |
Unlike a DFlash draft, this model reads the audio through cross-attention, the same way the target does. It
proposes one token per forward pass of its two decoder layers, with its own KV cache, and the target verifies the
proposals in one pass. Loaded as WhisperForCausalLM (the decoder alone), it works as the assistant in
transformers' assisted generation, which hands it the target's encoder output:
import torch
from transformers import WhisperForCausalLM, WhisperForConditionalGeneration
target = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", dtype=torch.bfloat16).cuda()
draft = WhisperForCausalLM.from_pretrained("wilsonzheng0327/Whisper-large-v3-distil-draft", dtype=torch.bfloat16).cuda()
tokens = target.generate(input_features, language="ja", task="transcribe", assistant_model=draft)
sglang-omni has no path for a separate Whisper draft yet.
<|startoftranscript|>, the language, <|transcribe|>,
<|notimestamps|>) and the transcript; the loss is cross-entropy on the transcript tokens and the end token.
English clips sent without a language use <|en|>, as sglang-omni does.Simulated greedy speculative decoding on CUDA (bf16): from each committed position the draft proposes 7 tokens
greedily, with large-v3's 88 suppress_tokens masked, the longest prefix matching the target's transcript is
accepted, and the target adds one token; accept = generated tokens / rounds. Held-out audio only: LibriSpeech
test, FLEURS test, Common Voice ja and yue test, and the last two YODAS2 shards of each language, 3,051 clips with
a complete transcript.
| draft | Libri clean | Libri other | FLEURS en | YODAS en | FLEURS zh | YODAS zh | CV ja | FLEURS ja | YODAS ja | CV yue | FLEURS yue | en | zh | ja | yue | all |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| this draft | 6.57 | 6.15 | 6.04 | 4.44 | 5.97 | 5.46 | 5.50 | 6.36 | 5.25 | 5.12 | 5.22 | 5.50 | 5.67 | 5.61 | 5.19 | 5.53 |
distil-whisper/distil-large-v3 |
6.80 | 6.29 | 6.27 | 3.68 | 1.03 | 1.03 | 1.05 | 1.02 | 1.03 | 1.09 | 1.03 | 5.14 | 1.03 | 1.03 | 1.05 | 1.36 |
| a 5-layer DFlash draft on the same labels, which sees no audio (not published) | 3.25 | 3.11 | 3.18 | 3.08 | 3.39 | 3.49 | 3.51 | 3.78 | 3.56 | 3.22 | 2.85 | 3.15 | 3.45 | 3.62 | 2.95 | 3.33 |
On English it gains 0.36 over distil-large-v3, all of it on YODAS2. It is 0.14 to 0.23 lower on LibriSpeech and FLEURS, read speech where distil-large-v3 was already strong.
Accept length is not speed. This draft spends a decoder pass per proposed token, while a DFlash draft proposes its whole block in one pass.
Wall clock of target.generate(...) with transformers 5.19, bf16, SDPA, batch 1, on one H200: 40 held-out clips
per source, 440 in all, with each clip running all three modes in an order that rotates by clip. The assistant
settings are transformers' defaults. The time covers the whole request, encoder included, and the speedup is the
no-draft time over the assisted time.
| assistant | Libri clean | Libri other | FLEURS en | YODAS en | FLEURS zh | YODAS zh | CV ja | FLEURS ja | YODAS ja | CV yue | FLEURS yue | en | zh | ja | yue | all |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| this draft | 2.33 | 2.22 | 1.92 | 1.42 | 1.70 | 0.88 | 1.75 | 2.03 | 1.33 | 1.64 | 1.55 | 1.75 | 1.30 | 1.59 | 1.57 | 1.58 |
distil-whisper/distil-large-v3 |
2.51 | 2.07 | 2.26 | 0.97 | 0.58 | 0.49 | 0.62 | 0.61 | 0.53 | 0.62 | 0.60 | 1.44 | 0.54 | 0.57 | 0.61 | 0.73 |
begin_suppress_tokens on every model. sglang-omni does not mask them, and 400 of the 500
held-out Cantonese labels start with a bare space (token 220). transformers masks them on its plain path and not
on its assisted path, so with the default config the two paths decode different Cantonese transcripts.Apache-2.0. This model is a fine-tune of distil-whisper/distil-large-v3, which is released under the MIT license
(Copyright 2023 The OpenAI Authors and The HuggingFace Inc. team); its notice is kept in LICENSE-distil-large-v3.
distil-whisper/distil-large-v3