An int4 quant of Qwen/Qwen3.8-27B that I made to run on a single Intel Arc
Pro B70. It's asymmetric int4, group 128, GPTQ with an MSE clip search, stored in standard GPTQ v1 format
(sym: false). The vision tower, the MTP head and the lm_head stay BF16. Compared with a regular symmetric GPTQ int4
of the same size, it has about half the KL divergence from BF16.
Made by @imdariotoo. I post my B70 and local-model releases on X, so follow there for the next ones.
KL divergence over the top-20 logits on held-out 2048-token windows (lower is better):
| code | technical prose | chat (private samples) | all | top-1 agree | |
|---|---|---|---|---|---|
| SergiioB's GPTQ int4 (sym g128, GPTQModel) | 0.087 | 0.103 | 0.050 | 0.078 | 91.3% |
| this quant | 0.037 | 0.057 | 0.025 | 0.039 | 93.8% |
| this quant + fp8 lm_head (as served in the cookbook) | 0.038 | 0.058 | 0.025 | 0.039 | 93.5% |
In-head arithmetic, greedy, thinking off, same server:
The sum(i*i for i in range(4)) check (SergiioB reports every symmetric int4 export of this model answers 30 instead of 14; BF16, EXL3, OpenVINO int4 and Q4_K_M answer 14). Measured 2026-10-15 on one B70, served as in the cookbook:
Built for and tested on Intel Arc Pro B70 + vLLM XPU v0.31.0 (the v0.32.0 image runs the same code path; tested there with the Uncensored variant) with the launcher and B70 vLLM changes in riceharvest/b70-qwen38-cookbook. That setup gives 262K context with vision, MTP k=4, and fp8 KV on one card.
hf download darioooooo0o/Qwen3.8-27B-GPTQ-Int4-asym --local-dir qwen38-b70
MODEL_DIR=$PWD/qwen38-b70 ./serve.sh # from the cookbook repo
On a B70 at 275 W, one request at a time:
| Decode, short prompts | 66-119 tok/s greedy depending on content (35 ms per MTP verify step) |
| Decode at 65K / 100K context | ~72 / 65-67 tok/s greedy, 73-77 / 66-69 sampled (43 / 46 ms per step) |
| Prefill | 2042 tok/s at 5.7K, 1422 at 74K, 824 at 237K |
| Context | 262,144 tokens with vision; 237K-token needle retrieved |
Prebuilt image: ghcr.io/riceharvest/b70-vllm:v0.32.0-b70.7 (= :latest, vLLM v0.32.0) has the B70 vLLM changes and kernels
baked in: docker run -d --device /dev/dri -v /dev/dri:/dev/dri:ro --shm-size 8g -p 8000:8000 -v ~/models:/models -e MODEL=darioooooo0o/Qwen3.8-27B-GPTQ-Int4-asym ghcr.io/riceharvest/b70-vllm:latest.
Many concurrent requests: add --mamba-ssm-cache-dtype float16 (after the image name, or B70_GDN_STATE=fp16 ./serve.sh).
Qwen3.8 stores its linear-attention (GDN) state in fp32 by default, and with many requests that state is a large part of what
the GPU reads per step. Measured on Qwen3.6-35B-A3B, which has the same GDN layers: 64 concurrent users went from 49 to 42-45
ms per token, with decode quality at the fp32 noise floor and gsm8k 98/100 vs 94/100
(details). Not measured on this model yet.
The image (v0.31.0-b70.3) with this quant: 102 / 77 tok/s code / prose for one request, 0/32 corrupted outputs with 8 concurrent thinking-length requests.
Other runtimes: the format is standard asymmetric GPTQ v1, so GPTQModel / transformers / vLLM on CUDA should load it. I only tested on Intel XPU. Stock vLLM XPU (v0.31, v0.32) does not load asym GPTQ correctly without the cookbook's asym GPTQ change (upstream PR vllm#61731).
b70quant.py from the cookbook. GPTQ (damp 0.01, act-order off), asym uint4 with per-group
scale + zero, and a clip search over 11 ratios minimizing |err|^2.4. One layer at a time on the B70, ~1 h.in_proj_a/in_proj_b.Credits: Qwen team (base model), SergiioB (B70 vLLM cookbook this builds on).
Questions or issues: reply to @imdariotoo on X or open a discussion on this repo.
model-extra0.safetensors 5.1 GBmodel-extra1.safetensors 1.8 GBmodel-layer10.safetensors 200 MBmodel-layer12.safetensors 200 MBmodel-layer13.safetensors 200 MBmodel-layer14.safetensors 200 MBmodel-layer16.safetensors 200 MBmodel-layer17.safetensors 200 MBmodel-layer18.safetensors 200 MBmodel-layer20.safetensors 200 MBmodel-layer21.safetensors 200 MBmodel-layer22.safetensors 200 MBmodel-layer24.safetensors 200 MBmodel-layer25.safetensors 200 MBmodel-layer26.safetensors 200 MBmodel-layer28.safetensors 200 MBmodel-layer29.safetensors 200 MBmodel-layer30.safetensors 200 MBmodel-layer32.safetensors 200 MBmodel-layer33.safetensors 200 MBmodel-layer34.safetensors 200 MBmodel-layer36.safetensors 200 MBmodel-layer37.safetensors 200 MBmodel-layer38.safetensors 200 MBmodel-layer40.safetensors 200 MBmodel-layer41.safetensors 200 MBmodel-layer42.safetensors 200 MBmodel-layer44.safetensors 200 MBmodel-layer45.safetensors 200 MBmodel-layer46.safetensors 200 MBmodel-layer48.safetensors 200 MBmodel-layer49.safetensors 200 MBmodel-layer50.safetensors 200 MBmodel-layer52.safetensors 200 MBmodel-layer53.safetensors 200 MBmodel-layer54.safetensors 200 MBmodel-layer56.safetensors 200 MBmodel-layer57.safetensors 200 MBmodel-layer58.safetensors 200 MBmodel-layer60.safetensors 200 MBmodel-layer61.safetensors 200 MBmodel-layer62.safetensors 200 MBmodel-layer00.safetensors 200 MBmodel-layer01.safetensors 200 MBmodel-layer02.safetensors 200 MBmodel-layer04.safetensors 200 MBmodel-layer05.safetensors 200 MBmodel-layer06.safetensors 200 MBmodel-layer08.safetensors 200 MBmodel-layer09.safetensors 200 MBmodel-layer11.safetensors 193 MBmodel-layer15.safetensors 193 MBmodel-layer19.safetensors 193 MBmodel-layer23.safetensors 193 MBmodel-layer27.safetensors 193 MBmodel-layer31.safetensors 193 MBmodel-layer35.safetensors 193 MBmodel-layer39.safetensors 193 MBmodel-layer43.safetensors 193 MBmodel-layer47.safetensors 193 MBmodel-layer51.safetensors 193 MBmodel-layer55.safetensors 193 MBmodel-layer59.safetensors 193 MBmodel-layer63.safetensors 193 MBmodel-layer03.safetensors 193 MBmodel-layer07.safetensors 193 MBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo darioooooo0o/Qwen3.8-27B-GPTQ-Int4-asym ./model-folderQwen/Qwen3.8-27Bdocker run --rm -d --name lsv-cb-rh27-tp1-mtp4 --net=host --shm-size=32g --ipc=host --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/models:ro -v /models/bench/cb-20261023/session/cb-rh27-tp1-mtp4-c1/session/cache:/cache --env-file /models/bench/cb-20261023/session/cb-rh27-tp1-mtp4-c1/session/server-configs/cb-rh27-tp1-mtp4.env --entrypoint vllm ghcr.io/riceharvest/b70-vllm:v0.32.0-b70.7@sha256:79bb3725d8eb820a6b6fcdd3fd470a765fcdeecc6c63de34ba0d89d13816536d serve /models/Qwen3.8-27B-GPTQ-Int4-asym --served-model-name qwen38-27b-rh --tensor-parallel-size 1 --distributed-executor-backend mp --quantization gptq --dtype float16 --kv-cache-dtype fp8 --max-num-seqs 8 --max-num-batched-tokens 8192 --cpu-offload-gb 4 --cpu-offload-params embed_tokens --language-model-only --max-model-len 65536 --gpu-memory-utilization 0.93 --speculative-config \{\"method\":\"mtp\"\,\"num_speculative_tokens\":4\,\"draft_sample_method\":\"probabilistic\"\} --compilation-config \{\"cudagraph_capture_sizes\":\[1\,2\,4\,5\,8\,10\,15\,20\,25\,30\,35\,40\]\} --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --generation-config auto --enable-prompt-tokens-detailsView run on LocalMaxxing docker run --device /dev/dri -v <model>:/model --entrypoint bash ghcr.io/riceharvest/b70-vllm:v0.31.0-b70.6 -lc 'vllm serve /model --quantization gptq --dtype float16 --kv-cache-dtype fp8 --max-model-len 65536 --gpu-memory-utilization 0.80 --max-num-seqs 8 --max-num-batched-tokens 8192 --speculative-config {"method":"mtp","num_speculative_tokens":4} --compilation-config {"cudagraph_capture_sizes":[1,2,4,5,8,10,15,20,25,30,35,40]} --language-model-only --reasoning-parser qwen3' (image env: packed MTP verify attention, int4 draft MTP layer + 64K-vocab int4 draft head, fp8 target lm_head)View run on LocalMaxxing Median of 3 community runs on 2 setups, from LocalMaxxing.
Run it yourself: setup guide