GitHub repo · Technical blog · Model collection

Flow-Drafter-Qwen3.5-27B-tr

Speculative-decoding draft model for Qwen/Qwen3.5-27B (Chained-Flow, joint-VAE drafter), adapted to Turkish.

Lossless by construction: a proposed token is emitted only if the target itself would have emitted it. Outputs can still differ from unspeculated decode, because verifying K+1 positions at once changes the target's own fp16 reduction order — the observed divergence is comparable to the run-to-run difference between two unspeculated runs.

Warm-started from Flow-Drafter-Qwen3.5-27B (sha256 31baf9ad6a417b15eba6c896b367277b0696c4933036242900cfa1a6dbc5b072), not trained from scratch. Identify the parent by that content hash, not by commit — a card-only edit moves the commit without changing a byte of the weights. These weights are sha256 f58eb4fc5079ea82a36a808f0b4b0f2984066d3590484fe2db6972351333e242.

Adapted on ~12.95M tokens of Turkish teacher states generated by Qwen/Qwen3.5-27B itself, in non-thinking mode, at LR 1e-4 for 3 epochs.

Usage

pip install chained-flow

# CF_DRAFTER_DIR is the only variable you need; a bare repo id works and pulls shortlist.pt with it.
# K comes from num_speculative_tokens, not an env var.
# --max-num-seqs IS REQUIRED HERE: this is a hybrid model needing one Mamba cache block per decode
# sequence, and vLLM's default 1024 does not fit ("exceeds available Mamba cache blocks"). 4B/9B
# do not need it.
CF_DRAFTER_DIR=ytu-ce-cosmos/Flow-Drafter-Qwen3.5-27B-tr \
vllm serve Qwen/Qwen3.5-27B --async-scheduling --max-num-seqs 64 \
  --speculative-config '{"method":"custom_class","model":"chained_flow.vllm_plugin.flow_proposer.FlowDrafterProposer","num_speculative_tokens":5}'

curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3.5-27B",
  "messages": [{"role": "user", "content": "Merhaba, nasılsın?"}],
  "chat_template_kwargs": {"enable_thinking": false},
  "temperature": 0
}'

chat_template_kwargs: {"enable_thinking": false} is required — this drafter was trained on prompts carrying the <think>\n\n</think>\n\n prefix.

Served speedup

Every number below was measured under one condition: natural EOS (no ignore_eos), concurrency 1, greedy, chain proposer at K=5, async scheduling on for every arm including the baseline, 2 repeats, through vllm serve. Prompts: the chained-flow Turkish holdouts (bench_data_tr), 4 domains, 400 requests per arm, 0 errors. Aggregation is token-weighted pooled, not a mean of per-domain ratios. Under natural EOS the arms emit different token counts (169.1 tokens/request here, not a forced 256), so the rates are comparable but the token totals are not.

26.0 → 50.5 tok/s, +94% against unspeculated decode.

Unlike the 4B and 9B siblings the English parent is not a net loss here (1.070x) — at 26 tok/s the base step is slow enough that accept 1.33 clears break-even. Adapting takes it 1.07x → 1.94x.

The candidate shortlist ships with the model and is loaded automatically; the loader refuses to run against a different one.

Caveats

Turkish prompt sets only; this card does not characterise English behaviour. English accept was measured offline during selection and moved −0.06 overall / −0.09 on free-form domains.

Files

• model.safetensors — the drafter (flow experts, Markov head, path head, jointly-trained VAE)
• chained_flow_tree_config.json — architecture + loss config
• shortlist.pt — English+Turkish candidate shortlist (see above)
• vae/ — the base VAE checkpoint