GitHub repo · Technical blog · Model collection

Flow-Drafter-Qwen3.5-4B

Expanded-data revision of Flow-Drafter-4B — a speculative-decoding draft model for Qwen/Qwen3.5-4B (Chained-Flow, joint-VAE drafter).

What changed in v2: trained on a broader, diversity-weighted mix (~46k rows, 10 sources) adding multi-turn chat (UltraChat), diverse instructions (No-Robots), creative prose (WritingPrompts), summarization (CNN/DailyMail) and translation (OPUS) on top of the original math/code/STEM core, to lift acceptance on the low-predictability domains.

Lossless by construction: a proposed token is emitted only if the target itself would have emitted it. Outputs can still differ from unspeculated decode, because verifying K+1 positions at once changes the target's own fp16 reduction order — the observed divergence is comparable to the run-to-run difference between two unspeculated runs.

Usage

pip install chained-flow

CF_DRAFTER_DIR=ytu-ce-cosmos/Flow-Drafter-Qwen3.5-4B \
vllm serve Qwen/Qwen3.5-4B --async-scheduling \
  --speculative-config '{"method":"custom_class","model":"chained_flow.vllm_plugin.flow_proposer.FlowDrafterProposer","num_speculative_tokens":5}'

Measured under vLLM — acceptance and speedup, per domain

Condition: batch 1 (concurrency 1), chain proposer at K=5, greedy, fixed 256 output tokens per request (ignore_eos) so both arms do identical work, async scheduling on for every arm including the baseline, 3 repeats (pooled spread ≤0.1%). Benchmark: RedHatAI/speculator_benchmarks, 8 distinct domains × 25 prompts, one run per domain, never pooled into a single split. Harness: guidellm 0.6.0 driving vllm serve (vLLM 0.25.1). acceptance = 1 + num_accepted_tokens / num_drafts from vLLM's own Prometheus counters (bonus token included, so a non-speculative baseline is 1.00). speedup = chain tok/s ÷ base tok/s, prefill included in the denominator.

This drafter does not win everywhere. On 1 of the 8 domains the chain arm is a regression — translation 0.94x. Speculation costs a draft pass on every step, so where acceptance is low the draft does not pay for itself. The pooled figure already includes those losses; quote it rather than the best domain.

At 4B the base decode step is cheap, so the drafter has little headroom to buy back; this is the size where speculative decoding is hardest to justify. The 9B and 27B drafters return more.

CF_SPEC_MAX_BATCH ships on by default and disengages speculation above a decode batch of 4 at this size, holding a loaded server near parity rather than below it.

Files

• model.safetensors — the drafter (flow experts, Markov head, path head, jointly-trained VAE)
• chained_flow_tree_config.json — architecture + loss config
• vae/ — the base VAE checkpoint

Drafts all K future hidden states in one flow pass and the base model verifies them. Trained 2-GPU DDP.

Companion models: Flow-Drafter-Qwen3.5-9B, Flow-Drafter-Qwen3.5-27B.