Qwen3.8-27B · GSQ-RCO IQ2_XS-mtp · CHiPS-Abliterated-v1

A version of ISTA DASLab's Qwen3.8-27B-GSQ-RCO-IQ2_XS-mtp.gguf with its refusal behavior reduced.

• On a held-out set of harmful requests, refusals drop from 88.5% to 1%, the same level as the Heretic build of the same quantization.
• Loops, empty answers, long-task behavior, tool calling and perplexity stay at the level of the original file.
• The change is confined to a very small part of the model. All other tensors are byte-identical to the original file.
• The edit was made with CHiPS (Causal Head-isolated Projection Steering), explained in this article. The recipe is not published, and the article's measurements are on other models.

Files

The vision projector (mmproj-Qwen3.8-27B-BF16.gguf) is not included. The one in the original repository was not modified, but vision use has not been tested with this file.

Evaluation

All numbers compare three files of the same IQ2_XS-mtp quantization: the original, the Heretic build, and this model. The refusal evaluation and perplexity were measured on the released file itself. The long-task and tool-calling evaluation was run with the same edit applied at load time on the original file. The released file was checked against that setup: on 64 prompts, the first-token distribution differs by a KL divergence of 0.006 or less, the most likely first token is the same on 62 of them, and the refusal results are at the same level.

Refusals and failure modes

104 harmful and 100 harmless requests, thinking enabled.

An empty answer here means the reasoning did not finish within the 1,536-token limit. For this model, all such cases were long reasoning with little repetition, not loops.

Long tasks and tool calling

24 multi-step long-form tasks and 40 tool-calling tasks.

The one long task flagged as a loop is the same task on which the original file and the Heretic build also run out of tokens with an empty output.

Perplexity

Teacher-forced perplexity on small in-house reference texts (512-token chunks). These are not the wikitext2 / C4 / FineWeb-Edu numbers of the original card.

All differences are within 0.3%. The Heretic build is a separately quantized file, so its perplexity is not directly comparable and is not listed.

Evaluation setup

• llama.cpp (llama-server), thinking enabled, temperature 0.6, top_p 0.95, top_k 20, at most 1,536 new tokens, fixed seed, q8_0 KV cache.
• The harmful and harmless request sets are in English and Japanese and were held out from all tuning. They are not published.
• Refusal is detected by matching common refusal phrases (English and Japanese) in the final answer. A loop is heavy repetition of short character sequences at the end of the reasoning and answer.

Usage

hf download Kujira/Qwen3.8-27B-GSQ-RCO-IQ2_XS-mtp-CHiPS-Abliterated-v1 Qwen3.8-27B-GSQ-RCO-IQ2_XS-mtp-CHiPS-Abliterated-v1.gguf --local-dir .

llama-server -m Qwen3.8-27B-GSQ-RCO-IQ2_XS-mtp-CHiPS-Abliterated-v1.gguf -ngl 99 --jinja

The file is a standard GGUF and needs no adapter or special build. The sampling settings in the evaluation setup above are a good starting point.

Limitations

• Refusals were counted by phrase matching, and the answers were not reviewed by people. An answer that avoids the request without using a refusal phrase is not counted as a refusal.
• The evaluation sets are small: one request is about 1%. The remaining 1 to 3% of refusals change from run to run.
• Only single-turn requests without a system prompt were tested. Multi-turn conversations, system prompts, and rephrased or adversarial requests were not.
• Only the IQ2_XS-mtp build is provided, and vision was not tested.

Intended use

This model follows many requests that the original model refuses, and it can produce harmful, offensive or inaccurate content. It is meant for research and for users who understand this. You are responsible for how you use it, including compliance with applicable laws and the license below. Add your own safeguards before using it in anything that serves other people.

License

Apache-2.0, inherited from Qwen3.8-27B and the original quantized release.

Credits

• Base model: Qwen3.8-27B by the Qwen team.
• Quantization: GSQ-RCO GGUF by ISTA DASLab. If you use the quantization, please cite GSQ and RCO (see the original card below).

---

Original model card

The following is the model card of ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, reproduced for reference. The files, results and usage below refer to the original repository, not to this model.

Qwen3.8-27B · GSQ-RCO GGUFs

Non-uniform GGUF quantizations produced with GSQ and RCO, with a vision projector for multimodal use.

---

Overview

This repository provides GGUF quantizations of Qwen3.8-27B at four sizes, together with the model's vision projector (mmproj) for multimodal use. In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF and run unmodified in llama.cpp, Ollama, and LM Studio.

Method summary. GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget. Together they yield a non-uniform GGUF at the requested size.

Both methods were developed at the Deep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.

---

Available files

Files follow the convention <model>-GSQ-RCO-<type>.gguf, where the suffix names the quantization class; the table lists each file's true whole-file average bit-width. The mmproj file carries the vision encoder and projector at BF16; one copy serves all quantizations.

Each quantization also ships an optional -mtp build (about 0.35 GB larger) that carries the model's Multi-Token Prediction head for speculative decoding in llama.cpp. The weights are otherwise identical, so quality is unchanged.

The IQ3_S model is the task-lossless operating point: it matches the base model exactly on AIME25 (100.00) and LiveCodeBench v6 (85.71) and is within 0.51 points on GPQA-Diamond, at just over one fifth of the BF16 size.

---

Results

All models are evaluated against the BF16 base model and the Unsloth Dynamic (UD) quantizations of the same base model. We report perplexity on wikitext2, C4, and FineWeb-Edu, the average over five zero-shot tasks (arc_easy, arc_challenge, hellaswag, winogrande, piqa), recovery (zero-shot average relative to BF16), and three reasoning and generation benchmarks: AIME25, GPQA-Diamond, and LiveCodeBench v6. Sizes are those of the files as evaluated.

At 3.50 bpw, IQ3_S is task-lossless: it reproduces the base model exactly on AIME25 (100.00) and LiveCodeBench v6 (85.71) and trails it by 0.51 points on GPQA-Diamond, giving a task average of 91.70 against the base model's 91.87 (99.8%) at 11.8 GB, a 4.6x size reduction. Against UD-IQ3_S it leads by 3.33 points on AIME25 and 1.71 on LiveCodeBench while being 0.2 GB smaller, though UD holds GPQA-Diamond by 0.51. At 3.00 bpw the model already matches the base on AIME25 at 10.1 GB, and at matched file size (8.4 GB) IQ2_XS leads UD-IQ2_S by 10.00 points on AIME25, 8.59 on GPQA-Diamond, and 4.57 on LiveCodeBench v6.

---

Usage

llama.cpp

# download (requires: pip install -U "huggingface_hub[cli]")
hf download ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf --local-dir .

llama-cli -m Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf -p "Explain mixed-precision quantization." -ngl 99

Vision (multimodal)

hf download ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF mmproj-Qwen3.8-27B-BF16.gguf --local-dir .

llama-mtmd-cli -m Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf \
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf \
  --image photo.jpg -p "Describe this image."

The projector was converted directly from the base checkpoint and verified against these quantizations.

Ollama

ollama run hf.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF   # pick the file matching your memory budget

LM Studio

Search the repo name, then pick a GSQ-RCO-* build from the file list.

---

Quantization procedure

1. Per-tensor database. Each weight tensor is quantized at every candidate GGUF quantization type with GSQ, yielding a searchable database of quantized tensor variants.
2. RCO search. The budget-constrained Riemannian search assigns one quantization type per tensor such that the whole-file average bit-width meets the target.
3. Assembly. The selected per-tensor variants are stitched into a single standard GGUF file.

Reference implementations: GSQ at IST-DASLab/GSQ and RCO at IST-DASLab/RCO.

Reproducibility artifacts

Each released GGUF ships the files needed to audit how it was built:

The -mtp builds have their own allocation dumps; they list the same per-tensor assignment as the base model plus the 15 tensors of the MTP head.

---

Citation

If you use these models or methods, please cite both papers:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

---

Acknowledgements

We thank Verda and Scientific Computing at the Institute of Science and Technology Austria for providing the compute resources used to produce these models.

---

License

These quantized weights inherit the license of the base model (Qwen3.8-27B). The GSQ-RCO tooling is released by the Deep Algorithms and Systems Lab under its repository license.

_{Built with GSQ and RCO at the Deep Algorithms and Systems Lab · Institute of Science and Technology Austria}