Qwen3.8-27B 3-bit GSQ for capy-flash (one 16 GB GPU)
Ready-to-serve artifact of Qwen3.8-27B for capy-flash v0.7 or newer,
small enough for a single 16 GB card with 64k tokens of context, MTP drafting and vision. The weights are
ISTA-DASLab's 3-bit GSQ release imported verbatim (no re-quantization), plus the official MTP layer for drafting.
The same file also runs on two cards (--profile 27b).
Quality cost, measured — read this first. 3 bits are not free. Against the NVFP4 27B served by vLLM on a GB300 (thinking on, greedy, 8,192 tokens), GSM8K is on par (0.968 vs 0.968, 1,319 questions) but MMLU-Pro drops (0.69-0.71 vs 0.78, 252 questions; about 0.72 with capy-flash v0.7's empty-reply guards, which bring back the answers that stopped inside their reasoning), mostly because more answers run out of tokens while reasoning. If your machine fits the NVFP4 27B (two 16 GB cards, see the capy-flash README), that one is better.
What is inside
| part | format |
|---|---|
| text body (320 projections) | GSQ 3-bit, 128-code groups with FP16 scales (q3_g128_fp16), codes copied verbatim |
| output head / proposal head | GSQ 4-bit RTN words copied verbatim (q4_g64_fp16) |
| token embedding | the same 4-bit words re-expressed losslessly as Q8 (meant for --embedding-host) |
| MTP layer | official Qwen3.8-27B MTP (Q8 stem, NVFP4 MLP), as in our production 27B artifact |
| vision tower | capy-flash's official optional format |
Files: qwen3_8_27b_gsq3.ninfer (12,549,282,560 bytes) and its converter report. Check with sha256sum -c SHA256SUMS.
Run
Requires capy-flash v0.7.0 or newer, Linux (or WSL2 / Docker on Windows), CUDA 13.1, one sm_120 GPU with 16 GB of VRAM (RTX 50 series; 16 GB is the minimum). No special RAM or CPU needs: those are for Flash-Next.
hf download Feyd89/Qwen3.8-27B-GSQ-3bit-capy-flash --local-dir qwen3.8-27b-3bit
cd qwen3.8-27b-3bit && sha256sum -c SHA256SUMS && cd ..
CUDA_VISIBLE_DEVICES=0 ninfer-serve qwen3.8-27b-3bit/qwen3_8_27b_gsq3.ninfer --profile 27b-16gb --model-id qwen27b
--profile 27b-16gb: int8 KV, 65,536 tokens, MTP with 3 drafts, vision, embedding in RAM; 13.9 GB of VRAM in use.
Chat template updated on 23/10/2026: this artifact now embeds the maintained NInfer Qwen3.8 template (
developerrole, Responsesinstructions, system turns after the first message: what Codex CLI sends); the weights are byte-identical. If you downloaded it earlier, runhf downloadagain (onlyqwen3_8_27b_gsq3.ninferand its last file change), or patch your copy in place withpython3 tools/replace_chat_template.pyfrom the capy-flash repository.
Measured with this artifact (capy-flash v0.7.0)
RTX 5070 Ti 16 GB (and 2x for TP2), Ryzen 9 9900X, real-text prompts, cold prefill, chunk 2048:
one GPU (--profile 27b-16gb) |
two GPUs (--profile 27b) |
|
|---|---|---|
| decode with MTP, 8k real-text prompt, full clocks | 139 tok/s | 230 tok/s |
| prefill 4k / 16k / 64k | 3,401 / 3,211 / — tok/s | 5,787 / 5,567 / 4,304 tok/s |
At eco clocks (2026-10-19) the one-card decode was 105 tok/s where llama.cpp with ISTA's GSQ-RCO IQ3_S and its MTP drafting, same card and prompt, did 66-67.
Provenance
- Source:
ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ(compressed-tensors pack-quantized); MTP layer from the official BF16 Qwen3.8-27B release. - Converter: capy-flash
python -m tools.convert --recipe qwen3_8_27b_gsq3(artifact idcd9b64f5314945e3a523cbc7e1d2f379, 1,012 objects, 12,548,946,688 payload bytes). - The 3-bit layout and kernel routes follow roofkid/ninfer-4080 (Apache-2.0); see capy-flash's NOTICE.
License
Apache-2.0, like ISTA-DASLab's quantization and Qwen3.8-27B (LICENSE-Apache-2.0.txt). Attribution in NOTICE. Not affiliated with Qwen or ISTA.
LocalMaxxing