Qwen3.5-2B-DSpark

A DSpark draft model for speculative decoding with Qwen/Qwen3.5-2B as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. At temperature 0, mean acceptance length is 3.89 tokens committed per verification step, up to 5.40 on math_reasoning. Compared with the verifier alone, throughput is 2.00× higher on average, up to 2.88× on math_reasoning and 2.62× on HumanEval, and it is faster than the verifier's native MTP drafter on eight of the nine benchmark subsets.

Training code: rasyosef/train-dspark-draft-models.

Trained on 100,000 samples.

Usage

vLLM

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Qwen3.5-2B-DSpark --port 8000 \
  --gpu-memory-utilization 0.8 \
  --override-generation-config '{"temperature": 0}'

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.

llama.cpp

llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q8_0 \
  -hfd rasyosef/Qwen3.5-2B-DSpark-GGUF:Q8_0 \
  --spec-type draft-dspark --spec-draft-n-max 8 --spec-draft-n-min 0 \
  -fa on -ngl 99

The block size is read from the sidecar metadata (n-max is clamped to it). Speculative decoding is exact: the target verifies every proposed token, so greedy output equals the target alone; per-response timings report draft_n / draft_n_accepted.

Details

5 Qwen3 layers (hidden size 2048, intermediate size 6144, 16 attention heads over 8 KV heads, head dim 128, all layers sliding-window attention with a 2048-token window), ~0.5B params, bfloat16. Block size 8, draft vocabulary reduced to 50,000 from the verifier's 248,320 (selected by token frequency over the training data), aux hidden-state layers 1/6/11/16/21, confidence head with Markov (vanilla, rank 256).

Trained for 4 epochs at lr 4e-4 (AdamW, cosine schedule with 4% warmup) on 100,000 filtered Open PerfectBlend prompts with responses regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Samples prepared at 2048 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front.

Evaluation

Drafter results are at temperature 0 (greedy decoding), from evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8.

Acceptance

Weighted across all subsets: 3.892 over 120,525 verification steps. math_reasoning and HumanEval are highest, and both still accept more than one draft in five at pos_7, as does qa. summarization and translation are lowest and fall below 1% by pos_7.

Throughput (tokens/s)

Output throughput with a single concurrent request on one A100. Baseline is Qwen3.5-2B with no speculative decoding. MTP is the verifier's native MTP drafter, run at 4 and 8 draft tokens. The fastest configuration in each row is bolded. The last column gives DSpark's throughput divided by baseline throughput for each subset.

DSpark has the highest average speedup at 2.00× (unweighted mean of per-subset speedups), ahead of MTP at 4 draft tokens (1.57×) and at 8 (1.31×), and it is the fastest configuration on eight of the nine subsets. The gains are largest on math_reasoning (2.88×), HumanEval (2.62×) and tool_call (2.50×), and smallest on summarization (1.44×) and translation (1.12×). Translation is the one subset where MTP at 4 draft tokens is ahead (196.6 against 188.1); it is also the smallest subset, with fewer than 900 verification steps in each run.

MTP at 8 draft tokens is slower than MTP at 4 on every subset except tool_call, and it only matches or falls below the baseline on question, writing, summarization and translation.

Acceptance comparison with MTP (4 and 8 draft tokens)

Acceptance length for DSpark and for the verifier's native MTP drafter on the same subsets at temperature 0. The highest value in each row is bolded and named in the last column. At 4 draft tokens the ceiling on acceptance_length is 5.0 rather than 9.0.

MTP at 8 draft tokens has the highest acceptance length on eight of the nine subsets and on the weighted average. Its lead over DSpark is widest on rag (4.844 against 3.986) and translation (3.115 against 2.527), and narrowest on question (3.379 against 3.299). DSpark is highest only on math_reasoning (5.405 against 4.743). MTP at 4 draft tokens is below MTP at 8 on every subset. DSpark is above it on seven subsets and below it on summarization and translation.

DSpark is faster than MTP at 8 draft tokens even on the eight subsets where MTP accepts more tokens per step, so each of its draft-and-verify steps takes less time.

Limitations

Works only with Qwen3.5-2B and is not usable as a standalone model. The verifier is vision-language, but the drafter was trained on text-only prompts and was evaluated on text-only benchmarks, so acceptance on image inputs is untested. All results are for greedy decoding with a single concurrent request; acceptance with sampling is typically lower, and throughput under load is not reported. The verifier's native MTP drafter at 8 draft tokens has a higher acceptance length on eight of the nine subsets, though lower throughput on all nine. MTP at 4 draft tokens is faster than DSpark on translation, where DSpark's speedup over the verifier alone is smallest (1.12×). Acceptance at the last draft position is under 1% on summarization and translation and about 10% or less on rag, tool_call, question and writing, so a block size of 8 is partly wasted there. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Apache 2.0, inherited from the verifier. The speculators training code is Apache-2.0.