Gemma 4 31B-it DFlash2 speculator, block 16
RedHatAI's DFlash2 speculator
for Gemma 4 31B-it, fine-tuned to draft 15 tokens
a pass instead of 7 (block 16 instead of 8). Same architecture and the same weights layout as
RedHat's, in vLLM's speculators format
(config.json, config.py, model.safetensors); only block_size and the weights differ.
For the AMD RDNA4 build with the trunk quantised and this speculator merged in, see gemma4-31b-w4a8-dflash2-b16.
What it gains
On 119 held-out prompts (HumanEval, math reasoning, QA, writing) that training never saw, replayed with the target's own greedy replies (thinking off), every reply position an anchor. Tokens a step, the accepted drafts plus the trunk's own:
| speculator | chain | tree |
|---|---|---|
| RedHat's, block 8 | 5.28 | 5.89 (7 nodes) |
| RedHat's weights run at block 16, no training | 7.05 | 7.59 (15 nodes) |
| this one, block 16 | 7.48 | 7.96 (15 nodes) |
By category (chain): HumanEval 10.52 against block 8's 6.76, math reasoning 10.15 against 6.66, QA 3.27 against 3.16, writing 3.52 against 3.28. The first seven positions are not hurt: cut to 7 drafts it accepts 5.30, against 5.28 for the original.
Most of the step from 8 to 16 is free: RedHat's weights already draft positions 8 to 15 well, because DFlash2 drafts every position of the block at once. The fine-tuning adds 5 to 6% on top of that.
In radiance on four Radeon AI PRO R9700, verified as a 15-node tree, it decodes 19% faster for one user than block 8's speculator verified as a 7-node tree (24% on two cards). It is level to slightly ahead (+0.5 to +3%) from 2 to 16 users, where each tree is cut to 7 nodes. Details are on the radiance build's card.
How it was trained
The block extension follows DBloom: start from the trained block-8 speculator with block_size
set to 16, and weight the loss on the new positions (8 to 15) three times.
- Data: 15,600 prompts with no answers, drawn from OpenMathInstruct-2 (5,400), rStar-Coder synthetic SFT (3,700), self-oss-instruct-sc2 (2,700) and UltraChat 200k (3,800). Each prompt was answered by the target itself: Gemma 4 31B-it at W4A8 in radiance, greedy, thinking off, up to 2,048 tokens, sequences cut at 3,072 tokens. The target's hidden states at the speculator's five taps (layers 1, 17, 29, 47, 59) and its last layer were read out of radiance's prefill of those sequences, and stored at fp8 with a scale per token and layer.
- Trainer: speculators at commit
442ce1bf(the one RedHat trained with), FSDP2 over four Radeon AI PRO R9700 under ROCm. ROCm needed a few changes, all switched by environment variables: an eagerly built block mask, SDPA for the speculator's attention, the eager loss, bf16 gradient reduction, and no resharding after the forward. 64 anchors per 3,072-token pack. - Schedule: a pilot of the first 3,600 prompts (about 3,400 usable sequences), 8 epochs at learning rate 2e-5. Then the other 12,000, in four shards of 3,000, each trained 4 epochs at 1e-5 from the shard before.
The 12,000 more sequences moved the held-out numbers by less than 0.1%, from the pilot's 7.476 to 7.483 (chain). This recipe has plateaued, and this is its final checkpoint.
Use
- radiance: convert it into the trunk with
rad-convert google/gemma-4-31B-it --draft-model Ashlyn4202025/gemma-4-31B-it-speculator.dflash2-b16 --recipe gemma4-31b-dense-bf16-w4a8-df2.recipe --set quantization_config.quant_method=compressed-tensors, or download the converted file above. Block 16 pays with tree verification, which is in review as Perire/radiance#26 and #27 (block-16 trees). - vLLM: the format is speculators', and
speculative_tokensis 15. It was not tried with vLLM.
Limits
- Trained and measured with thinking off. With thinking on, its acceptance was not measured.
- Measured against Gemma 4 31B-it quantised to W4A8 (the target it was trained on). Against bf16 Gemma, the drafts should land as well, but that was not measured.
- Prose and open-ended QA gain little from the longer block: there, the trunk accepts about 2.5 drafts a step, at block 8 or 16.
Credits
- The speculator it was fine-tuned from: RedHatAI, trained with vLLM's speculators library. The DFlash2 architecture: z-lab.
- The block extension with a weighted tail: DBloom.
- Gemma 4 31B-it: Google.
License
Apache 2.0, the license of RedHatAI's speculator and of Gemma 4. The prompts' datasets carry their own licenses; only their prompts were used, and the replies are Gemma's.