Gemma 4 31B-it, W4A8, with RedHatAI's DFlash2 speculator (radiance)
Gemma 4 31B-it in one file for the radiance inference engine on AMD RDNA4 cards (Radeon AI PRO R9700, gfx1201), with RedHatAI's DFlash2 speculator merged in for speculative decoding:
- the trunk's linears at int4 (round-to-nearest with a per-group scale search, from Google's bf16 checkpoint), q, k and v as one linear, read against int8 activations;
- the lm_head at int8;
- the DFlash2 speculator (five layers on a 2048-token window, a grouped dynamic convolution, a candidate selector; block 8, 7 drafts a pass) at block fp8, and its draft head at int8;
- the bf16 vision tower.
| file | size |
|---|---|
gemma4-31b-dense-w4a8-df2-qkv.rad |
23.2 GiB |
gemma4-31b-dense-bf16-w4a8-df2.recipe |
the recipe it was converted with |
For tool calls, use the GPTQ version: gemma4-31b-w4a8-gptq-dflash2 is this file with its int4 trunk fitted by GPTQ, at the same size and speed. This file's rounded trunk can leave a streamed tool call stuck inside a string argument: BetterBench's order request stalled in 10 of 20 tries here, and in none on the GPTQ file.
Needs
radiance with Gemma 4 31B DFlash2 support, which is in review and not yet in a release:
Perire/radiance#21, stacked on
#2. The PRs above them make it faster (below), and
#25 adds the compose file for four cards. The image
perire2019/radiance:gemma4-31b-df is a build of the branch these PRs come from.
Quick start
On four cards, with the file in /srv/models:
hf download Ashlyn4202025/gemma4-31b-w4a8-dflash2 gemma4-31b-dense-w4a8-df2-qkv.rad --local-dir /srv/models
docker run -d --name radiance --network host \
--device /dev/kfd --device /dev/dri \
--group-add "$(getent group render | cut -d: -f3)" --group-add "$(getent group video | cut -d: -f3)" \
--security-opt seccomp=unconfined --ulimit memlock=-1 \
-v /srv/models:/models:ro perire2019/radiance:gemma4-31b-df \
--model /models/gemma4-31b-dense-w4a8-df2-qkv.rad --host :: --port 8000 \
--tp 4 --max-num-seqs 16 --max-model-len 65536 --kv-cache-dtype fp8 \
--num-speculative-tokens auto --max-num-batched-tokens 8192 \
--busy-prefill-chunk 512 --prefill-interleave 1
The server speaks the OpenAI API on port 8000. --num-speculative-tokens auto serves the depth the
speculator was trained at (7). With the radiance repository at
#25, deploy/compose/gemma4-31b-w4a8-df2-tp4.yaml
runs the same thing under Docker Compose.
Speed
Four Radeon AI PRO R9700 at --tp 4 (PCIe 4.0), the radiance bench's prompts, greedy except the
story. The same box measured main radiance's 31B with Google's MTP assistant head for comparison:
| code | prose | story, temperature 1 | decode after a 6442-token prompt | |
|---|---|---|---|---|
| this file, the whole PR series | 390 tok/s | 288 | 208 | 590 |
| this file, on the PR that adds DFlash2 | 277 | 204 | 148 | 436 |
| main's 31B with the MTP head | 209 | 164 | 148 | 233 |
- Prefill of that 6442-token prompt: about 5960 tok/s on the whole series.
- Several users at once (BetterBench, thinking off, 16 sequences), aggregate tok/s at 1 / 2 / 4 / 8 / 16 users: 302 / 512 / 658 / 835 / 981.
- 2.93 accepted tokens a step over seven 2-6K-token prompts.
How it was made
rad-convert google/gemma-4-31B-it --draft-model RedHatAI/gemma-4-31B-it-speculator.dflash2 \
--recipe gemma4-31b-dense-bf16-w4a8-df2.recipe \
--set quantization_config.quant_method=compressed-tensors -o gemma4-31b-dense-w4a8-df2-qkv.rad
The recipe is in this repository and in radiance's data/recipes/. No calibration data: the
trunk's quantisation is round-to-nearest with a scale search per 128-column group.
Quality
Not evaluated beyond what the PRs check: the engine's replies are byte-identical across the PRs that change kernels, 16 of 16 freshly started servers gave the same reply, and speculative decoding is exact (the trunk verifies every draft), so the speculator changes speed, never output.
Limits
- AMD RDNA4 only (radiance's kernels are built for gfx1201).
- Measured at
--tp 4only.
Credits
- Gemma 4 31B-it and its vision tower: Google.
- The DFlash2 speculator: RedHatAI, trained with vLLM's speculators library; the DFlash2 architecture: z-lab.
- radiance: StillDeadcode. Gemma 4 support in radiance: sfjes and keptn42.
License
Apache 2.0, the license of Gemma 4 and of RedHatAI's speculator.