Qwen3.8 27B rotated INT3 weights for Paiton on RDNA4

These are 3-bit replacement weights for the decoder projections of Qwen/Qwen3.8-27B, calibrated with our own GPTQ run on permissively licensed data. They are an add-on to the current Paiton Qwen3.8 release on one AMD Radeon AI PRO R9700, not a standalone checkpoint. The same target checkpoint supplies all other tensors, and the same DFlash2 drafter is used. Against the previous MXFP4 image, weighted decode rises 19.9 %, prefill rises 5–13 %, and model memory falls from 19.18 to 15.89 GiB. The cost was about 3 points of MMLU-Pro knowledge recall; the 12 October update recovers an estimated 0.5 to 1 point of it.

24 October 2026: second accuracy update. The group scales were distilled toward bf16 again, starting from the 12 October weights, on 4.2 million new tokens. The format, the 3-bit codes, the image and the launcher are unchanged; only scale values differ, and the speed is the same. Against the 12 October weights, the served KL divergence to bf16 on the R9700 falls by 3.0 %, the gap in next-token log-likelihood to bf16 by 17–22 %, and long documents (8K to 131K tokens) come 1–2 % closer to bf16 in every length bucket (details). The 12 October weights stay available at revision 3ef7eb149ec0bd9a0f7927491bf30f53f54ced01.

12 October 2026: accuracy update. The BF16 group scales were distilled toward the bf16 model. The format, the 3-bit codes, the runtime image and the launcher are unchanged; only scale values differ. Against the previous weights, the KL divergence to bf16 falls by 20.9 % (decode) and 23.8 % (prefill), and by 19.5 % when served on the R9700, at unchanged speed (details). The previous weights stay available at revision 7a2e3d702144f744c1c5a46eaa42c7a2309d5cf7.

GPU: Radeon AI PRO R9700 (gfx1201, 32 GB) · Size: 9.55 GB · Context: 65,536 by default, up to 262,144 (524,288 opt-in) · Cache: up to 628,877 tokens, 555,128 with images · Text weights

Requirements

Use exactly these revisions. The rotated Gated DeltaNet b/a rows were derived from the target revision's bytes.

Quick start

git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin

export PAITON_TARGET_DIR="$PWD/model-cache/qwen38-nvfp4"
export PAITON_DRAFT_DIR="$PWD/model-cache/qwen38-dflash2"
export PAITON_W3ROT_DIR="$PWD/model-cache/qwen38-w3rot-int3"
export PAITON_CACHE_DIR="$PWD/runtime-cache/qwen38-rocm10-20261005"
mkdir -p "$PAITON_TARGET_DIR" "$PAITON_DRAFT_DIR" "$PAITON_W3ROT_DIR" "$PAITON_CACHE_DIR"

hf download unsloth/Qwen3.8-27B-NVFP4 \
  --revision f0b7c9e722f5565102fff8481c99e4d86ae099c7 --local-dir "$PAITON_TARGET_DIR"
hf download tcclaviger/Qwen3.8-27B-DFlash2-FP8 \
  --revision ee0cb26a8279b7910cc28d82a8a3e15e4728d56f --local-dir "$PAITON_DRAFT_DIR"
hf download EliovpAI/Qwen3.8-27B-W3Rot-INT3-Paiton-RDNA4 \
  --revision 383fa4b221a992fc2437500328056ef05355c8eb --local-dir "$PAITON_W3ROT_DIR"
(cd "$PAITON_W3ROT_DIR" && sha256sum -c SHA256SUMS)

bash models/Qwen3.8-MXFP4-DFlash2/run-3bit.sh

Already cloned it? Run git pull first; the launcher pins the image above. run-3bit.sh serves these 3-bit weights from PAITON_W3ROT_DIR, and run-mxfp4.sh serves the MXFP4 path on the same image. Pick one mode by adding its flag, e.g. bash models/Qwen3.8-MXFP4-DFlash2/run-3bit.sh --mode long:

What you get with these weights (one R9700; all commands start with bash models/Qwen3.8-MXFP4-DFlash2/):

Every mode takes up to eight requests at once and uses the checkpoint's FP8 output head (since the 11 October launcher update). --mode long-kv4 keeps the embedding table (2.4 GiB) in pinned system RAM, uses a 4-bit KV cache, and reads long text-only prompts in 2,048-token steps so short questions from other users are answered meanwhile (about 0.7 s instead of 5.9 s during a 64K read). --mode long-512k applies the model's official long-context scaling to every request. Images up to 4K are read at full resolution. The plugin README has the same table for the MXFP4 path (run-mxfp4.sh), what each row gives up, and the measurements. Earlier images stay runnable with --image; the plugin README lists their exact references. Once the server is ready, it accepts OpenAI-compatible requests at http://127.0.0.1:18982/v1 with API model name Qwen3.8.

Method

• Weights. The MLP, attention and Gated DeltaNet projections of all 64 layers (24.3 billion weights) are symmetric INT3 with one BF16 scale per 128 input elements (3.125 bits per weight).
• Rotation. MLP gate/up, attention q/k/v and Gated DeltaNet in_proj are stored in a block-diagonal Hadamard-128 basis, and the runtime rotates their inputs to match.
• Calibration. We ran sequential GPTQ in the rotated basis with an MSE scale search on 292,864 tokens of permissively licensed data. CALIBRATION.md lists the sources.
• Scale distillation (12 October 2026). Starting from the GPTQ result, the BF16 group scales of all 64 layers were trained end to end so that the 3-bit model's next-token distributions match bf16 Qwen3.8-27B: KL divergence on assistant and plain-text tokens, under both serving policies (FP8 activations for decode, 4-bit for prefill). Each scale could move by −18 % to +22 %, and no 3-bit code changed. The data were 5.16 million tokens (57 % code) from seven of the permissively licensed sources in CALIBRATION.md, decontaminated against the evaluation sets; the manifest's finetune block lists them with revisions and licenses.
• Second scale distillation (24 October 2026). The same objective, starting from the 12 October scales, on 4.2 million new tokens (about a third code; also math, chat, multilingual, science and textbook text) from the permissively licensed sources in CALIBRATION.md, none shared with the 12 October data and decontaminated against the evaluation sets. Each scale could again move by −18 % to +22 % from its 12 October value, and no 3-bit code changed.
• Prefill. The rotated projections take 4-bit activations, scaled per token and per 128-element group.
• Everywhere else. The other projections, and all projections during decode, take 8-bit (FP8) activations.
• Execution. Native RDNA4 kernels in the Paiton image run these weights. All other tensors come from the pinned target checkpoint.

Accuracy

Accuracy update, 24 October 2026

Each row pairs these weights with the 12 October weights (3ef7eb14) on identical inputs, with a 95 % bootstrap interval over documents, defined as in the 12 October table below. The R9700 rows come from the Paiton image served with run-3bit.sh --mode long-kv4.

• Knowledge (−8 %), chat (−6 %) and tool-calling agents (−6 %) gain most in decode KL; code is unchanged within noise (+1 %).
• Tasks. GSM8K 5-shot, HumanEval and the MMLU-Pro subset, with the same prompts and seeds as below, run on a bf16 copy of each set of weights (weight-only, no activation quantisation): 95.45 / 94.51 / 63.50 % against 95.53 / 93.90 / 62.86 % for the 12 October weights. No difference is significant (McNemar p 1.0 / 1.0 / 0.11).
• Speed is unchanged. Same image, launcher and kernels. With --mode long-kv4 on one R9700, BetterBench 0.6.0 weighted decode reads 169.1 against 169.3 tok/s, and 484.3 against 479.1 tok/s at eight requests. The decode step is 28.8–29.0 ms with either weights. In the default mode the weighted decode reads 155.5 against 155.9 tok/s, and 565.9 against 564.0 tok/s at eight requests. LocalMaxxing runs the same number of decode steps per second (about 34 at one request). Its single-prompt c1 figure moves with draft acceptance on that one prompt (116.7 against 125.5 tok/s in the median runs).

Accuracy update, 12 October 2026

Each row pairs the 12 October weights with the 26 September weights (7a2e3d70) on identical inputs, with a 95 % bootstrap interval over documents. KL is KL(bf16 ‖ 3-bit) of the next-token distribution, averaged over assistant and plain-text tokens; top-1 is how often the 3-bit model's most likely next token is also bf16's. The first rows reproduce the runtime's arithmetic under both activation policies on MI355X GPUs; the R9700 rows come from the Paiton image itself.

• No category gets worse under either policy. Chat, code and math gain the most (decode KL −41 %, −35 % and −22 %); knowledge questions (−1 % decode, −6 % prefill) and tool calls under prefill (−0.7 %) gain least, within noise.
• Long context. The gain is smaller at 131K tokens than on shorter texts, but every bucket improves, from the first 8K tokens to 128K. On the R9700, the served 131K check found the new weights better or equal in every bucket.
• MMLU-Pro. This training run scores +1.57 / +0.57 points (prefill / decode policy), a second run with another seed +0.57 / +0.79 points; the paired intervals include zero. We read this as about +0.5 to +1 point of the 2.9 to 3.9 points by which the previous weights trailed bf16 in this scoring.
• Speed is unchanged, as the format and kernels are the same. On one R9700 with the current launcher and image (--mode long-kv4), the decode step takes 28.7–28.9 ms and the first token 66 ms (p50) with either weights, and prefill from 2K to 64K tokens is within 0.3 %. BetterBench weighted decode reads 170.1 against 172.8 tok/s, and 482.2 against 481.9 tok/s at eight requests; the single-stream difference comes from speculative-decoding acceptance on replayed conversations, not from kernel speed.
• Outputs differ from the previous weights, greedy ones included. The GSM8K, HumanEval, needle and long-session results below were measured with the 26 September weights and were not re-run for this update.

Previous weights, 26 September 2026

These results were measured in-model on 2026-09-26 with the Paiton image and the previous weights, using greedy decoding with thinking off. Each benchmark is paired against the MXFP4 image on identical items. Δ is in points with a normal-approximation 95 % CI, and p is the exact McNemar test.

DFlash2 mean acceptance length relative to MXFP4 changes by +0.1 % on 38 chat prompts, −0.4 % on GSM8K, +2.8 % on HumanEval, +1.8 % on MMLU-Pro and +2.0 % on the needle test.

Math, code and long-context retrieval stay within noise. Knowledge recall drops by about 3 MMLU-Pro points, which is the trade-off of 3-bit weights. The MXFP4 path (run-mxfp4.sh) remains the choice for maximum knowledge accuracy.

Long coding sessions. In 20-turn coding sessions that grow from about 100K to 181K tokens, with a new HumanEval+ task every turn (8 sessions, 160 tasks), --mode long-kv4 with the previous weights solved 87.5 % of the tasks with no decline from the first turns to the last (change per turn 0.00, 95 % interval −0.71 to +0.64), against 83.75 % for MXFP4 in --mode long. Cached turns started in 3.1 to 4.1 s and decoded code at about 210 tok/s (full table).

Performance

The benchmark is BetterBench 0.6.0 (quick preset) with the 65K release profile on one R9700, using two fresh server processes per image. The reference is the previous MXFP4 image, qwen38-rocm10-vllm029-65k-20260924-r3. These runs used the earlier v3 calibration of these weights. Speed does not depend on the calibration data or on the 12 October scale update, because the tensor format and runtime are identical. Changes are computed from unrounded values.

Time to first token shortens by the same factors as prefill throughput. Model memory falls from 19.18 GiB to 15.89 GiB, and the launcher gives the freed VRAM to the KV cache: 250,578 cached tokens instead of 174,634, or 3.8 concurrent 65K-token requests instead of 2.7. Since 28 September, the 65K mode stores that cache in 4 bits by default with these weights: vLLM reports 393,216 tokens, and six 61K-token requests run together. Its accuracy and long-session checks are in the plugin reference.

Files and verification

The repository holds 64 layer shards (w3rot-L00.safetensors … w3rot-L63.safetensors, 1,296 tensors), w3rot-index.json, w3rot-manifest.json, CALIBRATION.md, LICENSE, THIRD_PARTY_NOTICES.md and SHA256SUMS. The manifest records the format, per-shard SHA-256, b/a source hashes and calibration provenance, including 33 dataset entries with revisions and licenses. Since the 12 October update it also records the new shards' SHA-256, weights_version and a finetune block: the base revision and manifest hash, the objective, the data sources with revisions and licenses, and per-shard counts of changed values. The 24 October update records the same block against the 12 October weights (31.9 million of 190.1 million scales changed, no 3-bit code), with the 12 October block kept as previous_finetune. Everything else in it is unchanged. The paths in its sources section refer to the calibration environment.

sha256sum -c SHA256SUMS
sha256sum w3rot-manifest.json   # 04ace57455f8f54b7cd330f0cfe6741e641a63bbeaf47f43ac58a037db2eaec4

The 12 October weights' manifest (revision 3ef7eb14) is 2d1f6c312a253397ad5c293d4d7a5dc53c6e50de64e58f05f4fa50d59d211f67; the 26 September weights' manifest (revision 7a2e3d70) is 335c5c834fb3f455203908d2b5aff8131aca2fe64ad73d59feddf32247d41a70.

Limitations

• The weights are lossy. With the previous weights, MMLU-Pro scored about 3 points below MXFP4 (the 12 October update recovers an estimated 0.5 to 1 point). Outputs differ from the MXFP4 path, and greedy generations can diverge.
• The served task benchmarks were measured with the 26 September weights. The 12 and 24 October updates were checked with KL to bf16 on held-out texts, long documents up to 131K tokens, MMLU-Pro log-likelihood and a served check on the R9700. The 24 October update also ran GSM8K, HumanEval and MMLU-Pro on a weight-only bf16 copy. Served GSM8K, HumanEval, needles and the long coding sessions were not re-run.
• The evaluation covers GSM8K, HumanEval, a 1,400-question MMLU-Pro subset, 38 chat prompts and needle retrieval at 61,440 tokens. Multilingual and tool-calling quality are not measured. For the long modes, the plugin reports GSM8K, HumanEval and MMLU-Pro with every question placed after a 257K-token document in --mode long (95.83 / 91.46 / 60.29 against 95.30 / 93.90 / 59.71 at short context, within the paired 95 % intervals), and, in --mode long-kv4, that the 4-bit cache predicts five 131K-token documents as well as the FP8 cache. With prefix caching (--mode long-kv4), GSM8K, HumanEval, MMLU-Pro and the 61,440-token needle stay within noise of the 2 October mode; --mode long-512k stays within noise of --mode long-kv4 on the same tests and found 4 of 4 planted facts at 300K and at 500K tokens (long-mode results).
• The 5 October image serves these weights in all four modes. They cover the language model; with --vision (not in --mode long-512k), image input uses the target checkpoint's vision encoder. One R9700 only; Transformers, stock vLLM and llama.cpp cannot load them.
• The default 65K mode uses the 4-bit KV cache. The long modes use the FP8 cache (--mode long) or the 4-bit cache (--mode long-kv4 and --mode long-512k), all three with prefix caching. The MXFP4 path and other settings use the FP8 cache.
• --mode long-kv4 reads a new long prompt about 3 % slower than without prefix caching (7 to 13 % for short prompts) and keeps 2.4 GiB of system RAM pinned; a host that cannot pin it gets a 451,879-token cache instead. --mode long-512k is opt-in: its long-context scaling applies to every request, it needs that system RAM, and a 500K-token document takes about 6 minutes to read the first time.
• The weights are tied to the pinned target revision and to Paiton's rotation and activation formats.

License and attribution

These weights are released under Apache-2.0. They are a quantized derivative of Qwen/Qwen3.8-27B (Apache-2.0), and the rotated b/a rows derive from unsloth/Qwen3.8-27B-NVFP4 (Apache-2.0). The method builds on GPTQ and GSQ (Apache-2.0). The target checkpoint and the DFlash2 drafter (tcclaviger/Qwen3.8-27B-DFlash2-FP8, Apache-2.0) are not redistributed here. The calibration and scale-distillation data require these attributions:

• QASC. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen and Ashish Sabharwal (Allen Institute for AI), allenai/qasc. It is licensed under CC BY 4.0 and provided as is. We reformatted it into calibration prompts, and it is not redistributed.
• ODC-By datasets. The calibration contains information from FineWeb-Edu, FineWeb-2 and FinePDFs-Edu (Hugging Face), and the scale distillation contains information from FineWeb-Edu. They are made available under the ODC Attribution License 1.0, and their use is also subject to Common Crawl's Terms of Use.

THIRD_PARTY_NOTICES.md has the full notices · Paiton · Public Paiton plugin