FastFlowLM build: Qwen3.8-27B-DFlash2-NPU2
The DFlash 2 speculative-decoding draft of Qwen3.8-27B (z-lab/Qwen3.8-27B-DFlash2,
a mirror of incoai/Qwen3.8-27B-DFlash2), converted for the AMD Ryzen AI NPU and run by
FastFlowLM (flm). It does nothing on its own: it is the second half of the
qwen3.8-dflash2:27b tag, whose main weights are the existing qwen3.8-mtp:27b package (Qwen3.8-27B-NPU2, unchanged).
flm pull qwen3.8-dflash2:27b downloads both.
flm run qwen3.8-dflash2:27b
Needs FastFlowLM 1.0.12 or later. FLM_DFLASH=0 loads the model's own MTP head instead (the tag then decodes like qwen3.8-mtp:27b).
| Upstream | z-lab/Qwen3.8-27B-DFlash2 (BF16 safetensors, the source of the weights here), pinned at revision 50307d4c |
| Requires | qwen3.8-mtp:27b (main model, tokenizer, chat template, vision tower, kernels) |
| License | Apache-2.0 (LICENSE); the main model's license applies to the main weights |
| Draft | 5 Qwen3-style layers, hidden 5120, FFN 17408, 32 query / 8 key-value heads of 128, sliding window 2048, block of 8 tokens, two-tap dynamic convolutions, candidate selector (top 16, rank 256); 2 B parameters; shares the target's embedding table and output head |
| Weights | dflash_weight.q4nx, 1.39 GB: fc, q/k/v/o and MLP matrices in Q4_K (about 0.07 relative L2 per matrix); norms, the dynamic-convolution tensors, the selector projection and the two 248320 x 256 selector codebooks in BF16 |
| Taps | features of target layers 5, 19, 33, 47 and 61 (zero-based layer outputs), fused by the draft's fc layer |
How it is used
Each step the draft proposes 7 tokens in one pass and picks one coherent path through its top 16 candidates per position; the main model checks all 7 in one pass and keeps the longest prefix it agrees with, plus one token of its own. The draft runs on the same NPU kernels as the main model (no extra bitstream): its matrices on the fused Q4_K dequantize + matmul engine, its output projection on the 8-row language-model head, its attention, convolutions and selector on the CPU.
Measured draft quality
Teacher-forced replay of one run of the real model (NPU, greedy, 300 tokens, thinking off): for every step the draft is asked for 7 tokens and compared with what the main model actually emitted next. Mean number of the 7 drafted tokens kept per step (the model then adds one):
| Draft | code prompt (52 steps) | prose prompt (66 steps) |
|---|---|---|
| BF16 weights, fp32 math (z-lab's reference code) | 5.02 | 3.36 |
| this package's Q4_K weights, fp32 math | 4.87 | 3.49 |
| this package on the NPU (bf16 activations) | 4.60 | 3.41 |
Quantizing the draft costs nothing measurable beyond the noise of 50 to 70 steps; the bf16 activations of the NPU cost a few percent on code. The published DFlash 2 acceptance lengths (which count the added token) are 4.1 on MT-Bench and 4.4 to 5.5 on code and math; this engine keeps about 3.1 on open prose and about 6.6 on a short code prompt.
FastFlowLM measurements
Ryzen AI MAX+ box, Linux, default power mode, flm serve, 160 generated tokens, greedy, ten short prompts, decode tokens/s:
qwen3.8-mtp:27b |
qwen3.8-dflash2:27b |
|
|---|---|---|
| mean | 3.23 | 3.74 (1.16x) |
| per prompt | 0.87x to 1.47x |
The speedup depends on how predictable the text is (code and structured answers gain the most). One step takes about 1.2 s with either draft
(0.9 s to check the 8 positions, about 0.27 s to draft). The NPU checks the 8 positions in one pass, which rounds differently from decoding one
token at a time, so greedy output can differ from qwen3.8-mtp:27b at near-ties (4 of 5 short greedy answers were identical to it, 5 of 10
long ones). Drafted tokens are compared with the model's own most likely token and skip the sampler, as with the MTP head.
Credits
Draft model: Inco AI, DFlash 2 (blog); original DFlash: Chen, Liang, Liu (ICML 2026), z-lab/dflash.