Qwen3-ASR-0.6B DFlash draft
A DFlash draft for greedy speculative decoding with Qwen3-ASR-0.6B (Qwen/Qwen3-ASR-0.6B and
Qwen/Qwen3-ASR-0.6B-hf). The checkpoint layout and config fields are those of
mattymchen/Qwen3-ASR-1.7B-speculator.dflash (speculators DFlash): fc, hidden_norm, 5 Qwen3 layers,
norm, a reduced-vocabulary lm_head (frozen rows of the target's tied embedding) with d2t / t2d.
| setting | value |
|---|---|
| draft layers | 5 (hidden 1024, intermediate 3072, 16 heads / 8 KV heads, head_dim 128) |
| target aux layers | 2, 14, 25 (hidden states entering those layers) |
| block size | 8 (7 proposals per round, anchor = last committed token), causal within the block |
| mask token | 151705 |
| draft vocabulary | 48,000 (99.7% of the training label tokens); a 32,000 variant is in -32k |
| sliding window | 32,768 (trained on recordings up to 240 s) |
| precision | bf16 |
Training
- Audio: LibriSpeech train-clean-360, People's Speech (clean), VoxPopuli en, Emilia-YODAS en/zh/ja (CC BY; the CC BY-NC Emilia part is not used), AISHELL-1, Common Voice 17 ja, FLEURS en/zh/ja train: about 1,600 h; 51% English, 28% Chinese, 21% Japanese by clip.
- Labels: the target's own greedy transcripts (vLLM, empty system context, language auto), so the
output keeps Qwen's
language X<asr_text>prefix, as the runtime produces it. - Phase 1: speculators'
DFlashDraftModeland loss (cross-entropy with D-Pace position weights), AdamW lr 6e-4, 4% warmup, cosine, 10 epochs over 591,786 clips of 0.5-30 s, packed 16k-token batches, up to 3,072 anchors per batch, one H100. - Phase 2: 4 epochs on 23,928 long recordings (30-240 s spans of consecutive clips of one source, labelled whole) plus 150,000 short clips, from the phase 1 checkpoint, window raised to 32,768. Phase 1 alone collapsed on long audio (accept about 1.5 at 60 s).
Accept length
Simulated greedy speculative decoding on CUDA (bf16): from each anchor the draft proposes 7 tokens, the longest prefix matching the target's greedy transcript is accepted and the target adds one token; accept = generated tokens / rounds (the runtime's tokens / verify_rounds). Held-out clips only (FLEURS test, LibriSpeech test, Common Voice ja test, one unseen Emilia-YODAS shard per language); long sets join consecutive held-out clips to ~60 s and ~180 s. The reference row uses the 1.7B target and its own labels.
| draft | Libri clean | Libri other | FLEURS en | Emilia en | FLEURS zh | Emilia zh | CV ja | FLEURS ja | Emilia ja | en | zh | ja | all | 60 s en | 60 s zh | 60 s ja | 180 s en | 180 s zh | 180 s ja |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| this draft (48k vocab), Qwen3-ASR-0.6B | 6.36 | 5.83 | 5.64 | 5.86 | 5.34 | 5.26 | 4.27 | 4.67 | 4.04 | 5.95 | 5.29 | 4.33 | 5.19 | 6.18 | 5.50 | 4.46 | 6.21 | 5.57 | 4.44 |
| this draft, 32k vocab variant | 6.09 | 5.66 | 5.23 | 5.68 | 4.91 | 5.18 | 4.26 | 4.66 | 4.03 | 5.70 | 5.07 | 4.31 | 5.04 | 5.90 | 5.22 | 4.45 | 5.95 | 5.24 | 4.44 |
| phase 1 only (short clips), 0.6B | 6.43 | 5.86 | 5.59 | 5.88 | 5.25 | 5.30 | 4.26 | 4.58 | 4.03 | 5.97 | 5.28 | 4.29 | 5.17 | 1.30 | 1.38 | 1.36 | 1.15 | 1.19 | 1.19 |
| reference: mattymchen 1.7B draft on Qwen3-ASR-1.7B | 6.23 | 5.94 | 4.72 | 5.08 | 5.04 | 4.34 | 2.10 | 2.09 | 2.09 | 5.46 | 4.60 | 2.09 | 3.51 | 5.45 | 4.76 | 1.87 | 2.94 | 2.96 | 1.54 |