GGUF quants of Qwen/Qwen3.8-27B for llama.cpp, on CUDA, Metal or CPU.
Qwen3.8-27B-<QUANT>.gguf is one file per quant, with the tokenizer and chat template inside.mmproj-Qwen3.8-27B-F16.gguf (0.93 GB) and mmproj-Qwen3.8-27B-Q8_0.gguf (0.63 GB) are the vision projector, for use with any quant.| Quant | Size | Mean KL vs BF16 | Same top-1 vs BF16 |
|---|---|---|---|
| Q8_0 | 29.05 GB | 0.0009 | 97.82% |
| Q6_K | 22.43 GB | 0.0023 | 96.83% |
| Q5_K_M | 19.54 GB | 0.0047 | 95.44% |
| Q4_K_M | 16.56 GB | 0.0097 | 93.46% |
| Q4_K_S | 15.36 GB | 0.0161 | 92.12% |
| IQ4_XS-imat | 15.31 GB | 0.0195 | 91.65% |
| IQ4_XS | 15.42 GB | 0.0250 | 90.38% |
| Q3_K_M | 13.50 GB | 0.0545 | 86.75% |
| IQ3_M | 12.77 GB | 0.0614 | 85.95% |
| Q2_K | 10.86 GB | 0.1725 | 79.21% |
How we measured:
| File | Size | Mean KL vs BF16 | Mean KL on chat text | Same top-1 vs BF16 |
|---|---|---|---|---|
| Xylolabs Q4_K_M | 16.56 GB | 0.0097 | 0.0162 | 93.46% |
| Unsloth UD-Q4_K_M | 16.56 GB | 0.0093 | 0.0185 | 93.95% |
| Xylolabs Q4_K_S | 15.36 GB | 0.0161 | 0.0265 | 92.12% |
| Unsloth UD-Q4_K_S | 15.36 GB | 0.0157 | 0.0267 | 92.73% |
All were measured with the same build, texts and reference. Chat text is 20,440 positions of chat-formatted English, Korean, code and math.
| Tensor class | Format |
|---|---|
| Weight matrices (attention, SSM and MLP projections, token embedding) | the quant type of the file; output.weight is Q6_K |
Norms, ssm_a, ssm_dt.bias, ssm_conv1d, q/k norms (360 tensors, 10.7 MB) |
F32 |
Use llama.cpp commit cd959bae99ff4bed22bef2604f45241c09bbb8c8 or later.
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout cd959bae99ff4bed22bef2604f45241c09bbb8c8
cmake -B build -DCMAKE_BUILD_TYPE=Release # add -DGGML_CUDA=ON for NVIDIA GPUs
cmake --build build -j --target llama-cli llama-server
./build/bin/llama-cli -m Qwen3.8-27B-Q5_K_M.gguf --jinja -c 8192 -ngl 99
Add --chat-template-kwargs '{"enable_thinking":false}' to answer directly, without the reasoning pass. llama-server takes the same flags.
Qwen/Qwen3.8-27B to a BF16 GGUF with convert_hf_to_gguf.py, and the vision projector with --mmproj.llama-imatrix on 60 chunks of an English and Korean calibration text.llama-quantize --imatrix qwen38-27b.imatrix <bf16 gguf> <out> <QUANT>.MIX-Q4_K_M.args, MIX-Q4_K_S.args in this repository), applied with --tensor-type-file, and a chat-text importance matrix.LICENSE (unchanged from the Qwen3.8-27B release).NOTICE names the source model and its copyright holder.Qwen/Qwen3.8-27B