Qwen3.8-27B GSQ-RCO IQ3_S MTP — ASCII Condensed

Ok, so the model cards never get to the point. This model is designed to get the most out of 16gb vram while still having a large context window, which Qwen3.8 desperately needs because it can't stop overthinking. Here is the move to maximize context:

• Use a headless linux server. By defauly llama.cpp reserves 1gb VRAM for the operating system. Find an old SSD, throw linux on there to dual boot into, and you only need to reserve like 64mb of VRAM.
• Offload mmproj vision to system RAM (mmproj vision file is on the original model page, though per note below, KV-streaming isnt currently compatible with vision)
• Offload the token embeddings to system RAM
• Use Q8 K-Cache, Q4 V-Cache. K-cache is upstream and has a greater impact on performance, so that is preserved. You could always just go Q4 KV-cache across the board and get more context and not build your own CUDA kernals but this should improve performance (note: requires compiling your own llama.cpp version in order to use mixed quant KV-cache, but it's built into the llama.cpp branch with KC cache streaming)
• Use the new KV-cache streaming (https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming). Two benefits: 1) if you have PCIe5 you can stream context window off system RAM with a lessoned performance impact, but PCIe4 is much slower. 2) More importantly, this allows you to pool the MTP and main KV-cache, saving footprint space and allowing you to still use the MTP, which I had removed for more context prior to the streaming release. (note: currently KV-cache streaming isn't compatible with vision)

This model is based on Qwen3.8-27B GSQ-RCO. To avoid letter salad, GSQ-RCO is just the quant method ISTA DASLab used to make a dynamic quant. Means the tensors are at different quant sizes. To my understanding, Unsloth makes dynamic quants by identifying which tensors break easiest and thus determine the dynamic tensors by which KLD worsen fastest when shrunk. ISTA DASLab took the approach of identifying which tensors break task results as they shrink and in theory better preserves the overall shape of the tensor relationships. Frankly to me it kinda sounds like shrinking the model by overfitting it to the benchmarks, but hey, people seem to really like the base GSQ_RCO model, most independent benchmarking has been decent, and if it works the size is very attractive. I'm sure someone will let me know if anything I've said here is misguided.

Testing it, but I have no benchmarks yet.

What changed

What I've done here is take Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf, and strip out the non-ascii characters unrelated to coding, using the tool created by bsaleh03 here: ASCII-Condensed Prune Tools. I used the "P1G" policy, so the chacters that are kept are ASCII, Math and Typography symbols, Greek, Box drawing, currency, units, status glyphs. All special tokens, all 256 byte-fallback tokens, and partial UTF-8 fragments remain. This saved about 500mb footprint.

"ASCII Condensed" describes the retained vocabulary; it is not a restriction on output characters. Other text can still be represented through byte fallback, potentially using more tokens. Changes to tokenization can affect generation and quality, especially for non-ASCII text.

Only vocabulary rows in token_embd.weight and output.weight were gathered, directly in their existing quantized representation. No dequantization, requantization, fine-tuning, or change to the other tensor payloads was performed. Special token IDs were remapped and BPE merges were filtered. The embedded chat template was preserved.

Download and run

While this repository is private, authenticate as an account with repository access:

hf auth login
hf download PlinyTheMedium/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp-ASCII-Condensed Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp-ASCII-Condensed.gguf --local-dir .

Example for a recent llama.cpp build that supports the qwen35 architecture:

llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp-ASCII-Condensed.gguf --jinja -ngl 99 -c 8192

The example command has not been exercised against this derivative. Choose GPU offload and context settings for your hardware; file size alone does not determine runtime memory use. MTP tensors are present, but speculative decoding depends on runtime support and configuration.

This repository includes the language-model GGUF. The upstream vision projector is not included, and multimodal use of the pruned derivative has not been tested. Refer to the upstream repository for its projector and original usage instructions.

Provenance and verification

• Base model: Qwen/Qwen3.8-27B.
• Quantized source: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, file Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf.
• Source SHA-256: 58fd826723939933dc86f45b7fe04545cbc2de1c70f6fe2cdd3858c87a98c12f. The local source checksum matches the upstream file.
• Pruning tool commit: 376a426d5c6b30f28e03c6efaa9edcf85048773f.
• Policy: --policy P1G, with no additional keep/drop overrides.
• Pruning dependencies: GGUF 0.19.0 and NumPy 2.5.3.
• Checksums for the released GGUF and supporting files are in SHA256SUMS.

verify-P1G.log records ALL CHECKS PASSED: exact P1G retained token set, valid tokenizer structure and remapped special IDs, preserved merge ordering, unchanged tensor types, and expected tensor shapes. All 864 non-vocabulary tensor payloads, including MTP tensors, are byte-identical to the source. 4,065 sampled retained rows in each vocabulary tensor match the source; vocabulary-row verification was sampled rather than exhaustive.

The original generation log prune-P1G.log records the output name ending in ASCII-P1G.gguf. That output was subsequently renamed to ASCII-Condensed.gguf; the release uses the latter name.

To reproduce, check out the pinned pruning tool, install the dependencies, and run:

python prune_vocab.py Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp-ASCII-Condensed.gguf --policy P1G
python verify_prune.py Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp-ASCII-Condensed.gguf --policy P1G

License and credits

The weights inherit the base model's Apache 2.0 license; a copy of its LICENSE is included. This modified release applies P1G vocabulary pruning as described above. Credit to Qwen for the base model, ISTA DASLab for the GSQ-RCO quantized source, and the bsaleh03 for the ASCII-Condensed Prune Tools. The source's attribution metadata is preserved in the GGUF.