Gemma 4 31B-it, W4A8 by GPTQ, with RedHatAI's DFlash2 speculator (radiance)
Gemma 4 31B-it in one file for the radiance inference engine on AMD RDNA4 cards (Radeon AI PRO R9700, gfx1201), with RedHatAI's DFlash2 speculator merged in for speculative decoding.
It is gemma4-31b-w4a8-dflash2 with its int4 trunk chosen by GPTQ instead of rounded to nearest. The grid, the kernels, the size and the speed are the same. The difference is that its streamed tool calls close their string arguments, as the bf16 model does.
- The trunk's linears are int4, fitted by GPTQ against the bf16 model's own activations (group 128, f16 scales with a scale search). q, k and v are one linear, read against int8 activations.
- The lm_head is int8.
- The DFlash2 speculator (block 8, 7 drafts a pass) is block fp8, and its draft head int8.
- The vision tower is bf16.
| file | size |
|---|---|
gemma4-31b-dense-w4a8g-df2-qkv.rad |
23.2 GiB |
gemma4-31b-dense-bf16-w4a8g-df2.recipe |
the recipe it was converted with |
Why GPTQ
Rounded to nearest, the trunk can leave a streamed tool call stuck inside a string argument. After
sku:<|"|>MS-110 in a call, the bf16 model closes the string (<|"|>, probability ~1.0). The
rounded trunk prefers **, (58% against 41%). When it takes that, the model writes on inside the
string until max_tokens, and the client never sees the call end. BetterBench's order request
stalled like this in 10 of 20 tries.
By GPTQ, the close is ~100%, and none of 40 tries stalled.
Needs
radiance 1.12.0 or later: the stilldeadcode/radiance
image. This is the same kind of container as the round-to-nearest one, so nothing else is needed.
Tree verification, in review at Perire/radiance#26,
makes it faster.
Quick start
On four cards, with the file in /srv/models:
hf download Ashlyn4202025/gemma4-31b-w4a8-gptq-dflash2 gemma4-31b-dense-w4a8g-df2-qkv.rad --local-dir /srv/models
docker run -d --name radiance --network host \
--device /dev/kfd --device /dev/dri \
--group-add "$(getent group render | cut -d: -f3)" --group-add "$(getent group video | cut -d: -f3)" \
--security-opt seccomp=unconfined --ulimit memlock=-1 \
-v /srv/models:/models:ro stilldeadcode/radiance:1.12.0 \
--model /models/gemma4-31b-dense-w4a8g-df2-qkv.rad --host :: --port 8000 \
--tp 4 --max-num-seqs 16 --max-model-len 65536 --kv-cache-dtype fp8 \
--num-speculative-tokens auto --max-num-batched-tokens 8192 \
--busy-prefill-chunk 512 --prefill-interleave 1
The server speaks the OpenAI API on port 8000. On two cards, use --tp 2. radiance's
deploy/compose/gemma4-31b-w4a8-df2-tp4.yaml serves the round-to-nearest file. To serve this one
under Docker Compose, change that file's --model line to this file's name.
Quality
Measured against the round-to-nearest file (RTN) and radiance's w8a8 container (Q8, 38.2 GB), with thinking off.
The stall. This file and RTN, both served by the stilldeadcode/radiance:1.12.0 image at
--tp 2. BetterBench's two order requests (a single call, and a round trip) were each streamed 20
times, sampled as BetterBench sends them:
| | <|"|> after MS-110 | **, | single calls that stalled | round trips that stalled |
|---|---|---|---|---|
| RTN | 41% | 58% | 10 of 20 | 1 of 20 |
| this file | ~100% | 0.04% | 0 of 20 | 0 of 20 |
KL against the bf16 model, on 44.5K held-out positions. The table counts only the model's own turns (31,975 positions). At system and user positions, the bf16 model's guesses are often odd, and any quantisation moves them. The bf16 reference is radiance's bf16 31B, a mean 0.02 nats from Hugging Face Transformers'.
| mean KL | median | p99 | top-1 | tool-call replies | |
|---|---|---|---|---|---|
| RTN | 0.341 | 0.0003 | 7.49 | 90.5% | 0.521 |
| this file | 0.251 | 0.0001 | 5.46 | 91.3% | 0.396 |
| Q8 | 0.226 | 0.0001 | 5.28 | 92.3% | 0.356 |
Over every position, the means are 0.988, 0.659 and 0.620. GPTQ closes 78% of the distance from RTN to Q8 on the model's turns. It is closer to bf16 than RTN on every source in the corpus: chat, code, math, reasoning and tool calls.
Tool use in BetterBench (tool_call and multi_turn, 1-16 users, sampled at 0.7, --tp 2).
Both files were served at once, then swapped between the card pairs for a second run. Both runs are
given:
| RTN | this file | |
|---|---|---|
| multi-turn: the right tool | 15/18, 15/18 | 18/18, 18/18 |
| tool calls: a valid call | 16/17, 15/17 | 17/17, 17/17 |
| tool calls: the right tool | 16/17, 15/17 | 17/17, 17/17 |
Speculative decoding is exact. The trunk verifies every draft, so the speculator changes the speed, never the output.
Speed
The speed is that of the round-to-nearest file: the same kernels, and the same 24.9 GB. The
speculator agrees with this trunk slightly more often. At 8 users, --tp 2, it averages 4.11 tokens
a step, against 3.96.
- BetterBench at 1-16 users,
--tp 2: level with RTN within the runs' noise (about 3%). - Decode on BetterBench's multi-turn conversations: 159.6 tok/s against 159.8.
- Four cards, with tree verification (#26): 381.2 tok/s single-stream decode, against 371.8 for RTN.
How it was made
The int4 codes were fitted to Gram matrices of the bf16 model's activations. radiance's Gemma 4
calibration tap
(Perire/radiance#33, in review) records them. It
serves Google's bf16 checkpoint and sums each dense linear's input over the corpus it is fed. Its
docs/GEMMA4-31B-GPTQ.md gives the full route.
- The Grams. The corpus went through the bf16 model at
--tp 4three times, 20 layers each time. Each pass took about 25 minutes, and the three wrote 45 GB of Grams. - The convert. With the Grams in place, the convert took 75 minutes on a 48-thread EPYC, using OpenBLAS:
CALIB=/calib rad-convert google/gemma-4-31B-it --draft-model RedHatAI/gemma-4-31B-it-speculator.dflash2 \
--recipe gemma4-31b-dense-bf16-w4a8g-df2.recipe \
--set quantization_config.quant_method=compressed-tensors -o gemma4-31b-dense-w4a8g-df2-qkv.rad
The corpus. 2.47M tokens in 4,252 whole conversations, rendered with the model's own chat template, thinking off:
| share | from |
|---|---|
| 30% tool calls | glaive-function-calling-v2, rendered as Gemma 4 tool calls and tool turns |
| 25% chat | UltraChat 200k prompts |
| 15% code | self-oss-instruct-sc2 prompts |
| 15% math | OpenMathInstruct-2 prompts |
| 15% reasoning | rStar-Coder prompts |
The chat, code, math and reasoning prompts carry the round-to-nearest 31B's own greedy replies. A held-out tenth was used only for the KL above. BetterBench's prompts, on which the stall was found and measured, are in neither part.
This file is a second convert, from the same corpus and recipe with the Grams recorded again; the first copy was lost. GPTQ's fit errors match the first convert's: 217 of the 240 weights agree to four digits, and the rest are within 0.1%. The KL, tool-use and speed numbers above were measured on the first convert, served by radiance 1.11.0. The stall table was measured on this file.
Limits
- AMD RDNA4 only (radiance's kernels are built for gfx1201).
- Measured at
--tp 2and--tp 4, on R9700 cards, with thinking off. - GPTQ protects what its corpus exercises. This corpus weights tool calls at 30% on purpose.
- keptn/gemma4-31b-w4a8 is also GPTQ (calibrated in Transformers), with Google's MTP head instead of a DFlash2 speculator.
Credits
- Gemma 4 31B-it and its vision tower: Google.
- The DFlash2 speculator: RedHatAI, trained with vLLM's speculators library. The DFlash2 architecture: z-lab.
- radiance: StillDeadcode. Gemma 4 support in radiance: sfjes and keptn42.
- Calibration prompts: glaive-function-calling-v2 (Glaive AI, Apache-2.0), UltraChat 200k (MIT), self-oss-instruct-sc2 (ODC-BY), OpenMathInstruct-2 (NVIDIA, CC-BY-4.0) and rStar-Coder (Microsoft, CC-BY-4.0). Only their Gram matrices went into this file.
License
Apache 2.0, the license of Gemma 4 and of RedHatAI's speculator.