MFD-D drafter for Qwen3-4B (1.42M prompts)
An MFD-D draft model for Qwen/Qwen3-4B, from Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative Decoding (arXiv:2611.13017; code: https://github.com/hhhhhhy5/multi-future-drafting; project page: https://hhhhhhy5.github.io/multi-future-drafting/).
MFD-D is an MFD drafter, a DFlash block drafter with K = 8 learned mode embeddings, with a default mode added. The same drafter without a mode embedding is also trained as a single-mode drafter. The SGLang patch of the code repository serves the 8-mode mixture tree at batch sizes 1-2 and the default path from 3 concurrent requests on, inside one engine. Decoding is lossless.
Model
| Target | Qwen/Qwen3-4B (no thinking) |
| Architecture | DFlash drafter: 5 layers of width 2560, reading target layers [1, 9, 17, 25, 33]; block of 16 (15 drafted positions); 8 mode embeddings (mode_embed, 8 x 2560) and a linear gate (mode_gate, unused at inference) |
| Parameters | 537M, bf16; the target's embedding and LM head are used at run time and are not stored here |
| Initialization | z-lab/Qwen3-4B-DFlash-b16 (MIT) |
| Training data | the 1.42M prompts of mlabonne/open-perfectblend (the data of Domino and DARTree), with responses regenerated by Qwen3-4B without thinking (T = 0.7, top-p 0.8, top-k 20, at most 2048 new tokens) |
| Training | 3 epochs, AdamW, learning rate 1e-4 (4% warmup, cosine), gradient clipping 1, sequences up to 3072 tokens, 36 sequences per step, 64 anchor blocks per sequence x 8 modes |
| Objective | relaxed winner-take-all over the 8 modes (the mode with the largest soft coverage of a block gets weight 0.9, the others 0.1 / 7), each mode trained with the D-PACE loss (smoothing 0.5); gate cross-entropy with weight 0.1; plus the single-mode D-PACE loss on the default path |
Results (from the paper)
SGLang 0.5.22 with the MFD patch, batch size 1, the same 5 tasks, B = 63, 64K draft vocabulary (Table 2):
| Method | T = 0 | T = 1 |
|---|---|---|
| DFlash chain (released drafter, SGLang native) | 4.06x (5.97) | 3.46x (5.10) |
| DDTree (released drafter) | 4.97x (7.81) | 4.44x (6.99) |
| D-PACE + DDTree (1.42M prompts) | 5.52x (8.72) | 4.72x (7.48) |
| This drafter | 5.89x (9.59) | 5.11x (8.27) |
Across batch sizes 1-32 (closed loop, 320 requests), its switch engine stays within 3.1% of the best fixed engine at every batch size (Figure 5 of the paper).
Usage
The drafter runs with the code of https://github.com/hhhhhhy5/multi-future-drafting. Hugging Face inference (from inference/hf, HF inference environment):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from model import DFlashDraftModel
from ddtree import ddtree_generate, enable_cpp_tree, enable_mode_attention
target = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", attn_implementation="sdpa", dtype=torch.bfloat16).cuda().eval()
draft = DFlashDraftModel.from_pretrained("hhhhhhy5/MFD-D-Qwen3-4B", attn_implementation="sdpa", dtype=torch.bfloat16).cuda().eval()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")
text = tok.apply_chat_template([{"role": "user", "content": "What is 17 * 23?"}], tokenize=False,
add_generation_prompt=True, enable_thinking=False)
ids = tok.encode(text, return_tensors="pt").cuda()
enable_cpp_tree(True, fused_logits=True)
enable_mode_attention(True)
out = ddtree_generate(model=draft, target=target, input_ids=ids, mask_token_id=draft.mask_token_id,
max_new_tokens=512, block_size=draft.block_size, tree_budget=256,
stop_token_ids=[tok.eos_token_id], temperature=0.0)
print(tok.decode(out.output_ids[0, ids.shape[1]:]))
SGLang serving (serving environment of the code repository):
bash inference/sglang/launch_server.sh --model Qwen/Qwen3-4B --draft hhhhhhy5/MFD-D-Qwen3-4B \
--draft-vocab inference/sglang/vocab/draft_vocab_qwen3-4b_64k.json \
--engine-config inference/sglang/configs/mfdd-switch-qwen3-4b.env --max-running-requests 32 --port 30000
Limitations
- The mixture is used at batch sizes 1-2 only; from 3 concurrent requests, the engine runs the single-mode default path.
- The drafter is tied to its target: it reads the target's hidden states and uses its embedding and LM head.
License and attribution
License: Apache-2.0. This drafter is fine-tuned from z-lab/Qwen3-4B-DFlash-b16 (MIT), whose notice applies to the weights it started from. The training responses were generated by Qwen/Qwen3-4B (Apache-2.0). Prompt source: mlabonne/open-perfectblend (Apache-2.0).
Citation
@article{huang2026mfd,
title={Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative Decoding},
author={Huang, Haoyu and Kang, Jialiang and Yang, Linlin and Yang, Xiao and Zhang, Baochang},
journal={arXiv preprint arXiv:2611.13017},
year={2026}
}