A DFlash2 draft for greedy speculative decoding with MOSS-Transcribe-Diarize 0.9B, in the speculators dflash2 format. It is warm-started from the published DFlash draft wilsonzheng0327/MOSS-Transcribe-Diarize-dflash at 2e3f00b093 and trained further with the same data and recipe. In simulated greedy decoding over held-out audio, it accepts 11.6% more tokens per round than that draft, and 8.1% more than a DFlash control trained the same way from the same start.
Served by sglang-omni on one H200, it is 1% to 6% faster than that draft at every load measured with synchronous scheduling, and as fast or up to 8% faster with overlapped scheduling, with the same CER and DER. The candidate selector adds work to every decode step, so the served gain is smaller than the accept gain. See Serving.
| setting | value |
|---|---|
| backbone | the DFlash draft's: fc, hidden_norm, 5 Qwen3 layers (hidden 1024, intermediate 3072, 16 heads / 8 KV heads, head_dim 128), norm |
| DFlash2 additions | grouped dynamic causal convolutions (kernel 2, group 16, identity-initialized) and a candidate selector (rank 256) that reranks the target head's top 16 per position |
| vocabulary | the full 151,936-token vocabulary; the draft has no lm_head of its own and uses the target's |
| target aux layers | 2, 14, 25 of the 28 decoder layers |
| block size | 8 (7 proposals per round, anchor = last committed token), causal within the block |
| mask token | 151705 |
| sliding window | 131,072 |
| trainable parameters | 162.5M (selector 78.1M, convolutions 2.66M) |
| weights | bfloat16 at the root, the serving copy; the float32 training export is in float32/ |
sglang-omni serves this draft with SGLang's DFlash2 draft model and DFLASH worker (SGLang 0.5.23):
sgl-omni serve \
--model-path OpenMOSS-Team/MOSS-Transcribe-Diarize \
--asr.engine.speculative_algorithm DFLASH \
--asr.engine.speculative_draft_model_path wilsonzheng0327/MOSS-Transcribe-Diarize-dflash2 \
--port 8000
The selector takes its top 16 with FlashInfer's top-k kernel. Without FlashInfer's prebuilt kernels (flashinfer-jit-cache), the first start compiles it, which took about 13 minutes on an H200 host; the sglang-omni Docker image ships it prebuilt.
Gain over the same server without a draft, on one H200 with the default server config and the prefix cache off. Each draft ran on its own GPU of one node, against its own server without a draft. movies800times is the mean of 3 passes per load and aishell4_long one pass over its first 8 recordings:
| Dataset | Draft | Scheduling | Concurrency 1 | 4 | 16 | 64 |
|---|---|---|---|---|---|---|
movies800times |
DFlash2 | Synchronous | 1.69x | 1.60x | 1.42x | 1.22x |
movies800times |
DFlash | Synchronous | 1.63x | 1.54x | 1.41x | 1.21x |
movies800times |
DFlash2 | Overlapped | 1.93x | 1.74x | 1.57x | 1.41x |
movies800times |
DFlash | Overlapped | 1.87x | 1.68x | 1.51x | 1.37x |
aishell4_long |
DFlash2 | Synchronous | 2.57x | 3.31x | 3.70x | |
aishell4_long |
DFlash | Synchronous | 2.50x | 3.15x | 3.51x | |
aishell4_long |
DFlash2 | Overlapped | 3.29x | 3.82x | 4.15x | |
aishell4_long |
DFlash | Overlapped | 3.13x | 3.58x | 3.85x |
In its own throughput this draft is 1% to 6% faster than the DFlash draft synchronously, 5% to 8% faster overlapped on aishell4_long, and within 1% overlapped on movies800times from 4 requests up. The logged acceptance length is 5.16 against 4.82 on movies800times and 5.80 against 5.28 on aishell4_long. CER and DER are unchanged, and no request failed. With FlashInfer's prebuilt kernels installed, a first start on an empty cache took 51 s with this draft and 45 s with the DFlash draft.
7ed657f. The checkpoint is the final step.Simulated greedy speculative decoding on CUDA (bf16 target), on the published draft's held-out sets: four unseen YODAS2 shards per language, AMI test, AISHELL-4 test and VoxConverse test.
Score is the mean over the seven length bins, and "all tokens" pools every generated token.
| draft | score | all tokens |
|---|---|---|
| published DFlash draft | 4.993 | 5.12 |
DFlash control (dflash-control/) |
5.155 | 5.29 |
| this draft, with the selector | 5.574 | 5.712 |
| this draft, without the selector | 5.307 | 5.448 |
By length:
| draft | 10 s | 60 s | 5 min | 20 min | 60 min | 90 min | meetings |
|---|---|---|---|---|---|---|---|
| published DFlash draft | 4.446 | 4.892 | 5.036 | 5.238 | 5.163 | 4.867 | 5.308 |
| DFlash control | 4.540 | 5.045 | 5.216 | 5.422 | 5.333 | 5.068 | 5.458 |
| this draft, with the selector | 4.886 | 5.491 | 5.630 | 5.867 | 5.773 | 5.573 | 5.797 |
| this draft, without the selector | 4.647 | 5.202 | 5.365 | 5.573 | 5.499 | 5.232 | 5.629 |
By language:
| draft | de | en | es | fr | ja | ko | ru | zh |
|---|---|---|---|---|---|---|---|---|
| published DFlash draft | 5.126 | 5.397 | 4.871 | 4.971 | 4.910 | 5.274 | 4.640 | 5.520 |
| DFlash control | 5.322 | 5.537 | 5.014 | 5.132 | 5.082 | 5.481 | 4.839 | 5.701 |
| this draft, with the selector | 5.785 | 5.878 | 5.388 | 5.584 | 5.503 | 6.012 | 5.372 | 6.065 |
| this draft, without the selector | 5.471 | 5.743 | 5.117 | 5.252 | 5.192 | 5.664 | 4.954 | 5.940 |
Per-set results, including slot-1 accuracy, are in eval/.
dflash-control/ is the published DFlash draft trained with the same data, recipe, steps and anchors on a second H200, with no DFlash2 additions. It keeps the 48,000-token head with d2t / t2d, in float32. It separates the gain from DFlash2 from the gain from training longer, and it is also a drop-in DFlash draft, though it has not been measured on a serving runtime either.
Apache-2.0, as the target model and the published DFlash draft.
float32/model.safetensors 650 MBdflash-control/model.safetensors 524 MBmodel.safetensors 325 MB