Qwen3.5-0.8B-abliterated, quantized

Fifteen GGUF files, measured against each other on one H200.

---

What this is

The abliterated checkpoint converted to GGUF and quantized from F16 down to ternary, with every file measured on the same prompt distribution and the same GPU.

Every number in the table below is measured, not estimated from bits per weight. That matters, because two of the fifteen formats turned out to be broken, and one of the "efficient" 2-bit formats is worse than a 4-bit file that is larger.

---

The table

WikiText-103 test set, 32 chunks of 512 tokens, llama-perplexity with -ngl 99. Speed from llama-bench, prompt 64 and generation 32, averaged over two runs.

* tokens per second measured on one NVIDIA H200 SXM (150 GB) with llama.cpp, full offload (-ngl 99), single stream. Other hardware will give different numbers; the comparison between quants is what transfers, not the absolute values.

---

What the numbers say

Q4_K_M is the knee. 5.7x smaller than BF16 for 4% worse perplexity. If you want one file, take that.

IQ4_XS is technically better than Q4_K_M: 20 MB smaller and 0.8 points closer to BF16. The difference is inside the noise of 32 chunks, but it is consistent across the two runs.

The cliff is at 4 bits, not at 2. Q3_K_M costs 26% and IQ3_M costs 9%. The gap between those two at nearly the same size is the largest surprise in the table: at 3 bits, the i-quant choice matters more than the file size.

Q2_K is worse than files that are larger and files that are smaller. 117 perplexity at 0.43 GB, against 27.6 at 0.40 GB for IQ2_M and 19.9 at 0.42 GB for IQ3_XXS. The old k-quant 2-bit recipe does not survive this architecture. Do not use it.

Ternary is broken here. Both TQ files produce 2.3 million perplexity and literal garbage:

TQ2_0: veniindowocaooth Weineraudiças如何通过aperevinartial womcanaision连云chendesia
Q4_K_M: The user is asking for the capital of France...

The prompt-processing speed tells the same story: 76 tok/s against 8,500 for every other format, a hundredfold fallback. llama.cpp has no fast ternary path for the Gated DeltaNet tensors of this model, and the result is unusable. The files are published for completeness and for anyone researching the format. They are not recommended for anything else.

Speed is flat. Every working format from 0.37 GB to 1.56 GB runs at 500 to 620 tok/s of generation. Those numbers are from one H200*; on a smaller GPU the absolute values drop, but the flat shape holds, because the model is bandwidth-bound at these sizes. The model is small enough that quantization buys memory, not throughput. If you are decoding a single stream on a modern GPU, the choice of quant is a quality decision, not a speed one.

Benchmarks per quant

Every file ran the same items: 200 MMLU questions, 200 HellaSwag items, 200 Winogrande items, 1000 GSM8K problems and 50 MATH-500 problems. Multiple choice is scored by generating a letter and parsing it strictly; GSM8K and MATH-500 by extracting the final answer.

* Same H200 as above. MMLU, HellaSwag and Winogrande use n=200, so a 4 point difference sits inside the noise. GSM8K uses n=1000 (±2.8 points) and MATH-500 n=50, where differences under 12 points mean nothing.

What the benchmarks add to the perplexity picture

Format adherence breaks before quality does. IQ3_XXS has a perfectly reasonable 19.96 perplexity, yet it returns a parseable letter only 14% of the time on MMLU and 16.5% on Winogrande. Perplexity cannot see this. For instruction-following use, that file is unusable even though its perplexity looks acceptable.

HellaSwag falls off a cliff at 4 bits. Q5_K_M scores 40.0%, Q4_K_M drops to 30.0%, and IQ4_XS to 21.5%. Perplexity called those three formats equivalent (15.88, 15.80 and 15.68). They are not equivalent on this task, and the Q5 to Q4 gap is the largest surprise in the report.

Q4_K_M beats BF16 on GSM8K. 338 problems solved out of 1000 against 290, Fisher p = 0.024. With n=1000 that is outside the noise, and it is still a decoding artifact rather than evidence that quantization improves arithmetic: greedy decoding on a weak model is sensitive to tiny weight changes, and the same file scores lower than BF16 on MMLU and HellaSwag. The honest reading is that at 29% accuracy the benchmark mostly measures which wrong answers the model commits to.

Two-bit quants keep pattern matching and lose reasoning. IQ2_M scores 53.5% on Winogrande and 26.5% on MMLU, close to its bigger siblings, but 4.7% on GSM8K and 0% on MATH-500. Multiple choice survives; anything requiring generated steps collapses.

The 1000-problem run corrected my earlier numbers. At n=100 GSM8K gave BF16 22% and Q4_K_M 31%, a 9 point inversion that looked like noise. At n=1000 the gap is 4.8 points and significant, which is a different claim. This is why the sample size matters more than the direction of the result.

Winogrande is near chance for every healthy file (48 to 56%, chance is 50%), which matches the base model's 58.5% on the standard harness with a different protocol. Here it works mainly as a format-failure detector.

---

How they were built

# 1. convert from the safetensors checkpoint
python convert_hf_to_gguf.py ./Qwen3.5-0.8B-abliterated --outtype f16 --outfile ablit-f16.gguf
python convert_hf_to_gguf.py ./Qwen3.5-0.8B-abliterated --outtype f16 --mmproj --outfile mmproj-f16.gguf

# 2. importance matrix for the i-quants
llama-imatrix -m ablit-f16.gguf -f corpus.txt -o imatrix.dat -c 512 --chunks 64

# 3. quantize
llama-quantize ablit-f16.gguf ablit-Q4_K_M.gguf Q4_K_M 16
llama-quantize --imatrix imatrix.dat ablit-f16.gguf ablit-IQ4_XS.gguf IQ4_XS 16

One detail worth knowing if you reproduce this: the very low bit i-quants fail with Missing importance matrix for tensor blk.24.attn_k.weight. Block 24 is the multi-token-prediction head, and it never runs during normal inference, so llama-imatrix collects no statistics for it. The fix is to pin that block to a higher precision:

llama-quantize --imatrix imatrix.dat --tensor-type 'blk\.24\..*=q8_0' \
    ablit-f16.gguf ablit-IQ2_M.gguf IQ2_M 16

The MTP block is 20M parameters. Keeping it at Q8_0 costs about 20 MB and lets the rest of the model go to 2 bits.

The importance matrix used here is included as imatrix.dat, computed from 64 chunks of Alpaca.

---

How to use

# single prompt
llama-cli -m Qwen3.5-0.8B-abliterated-Q4_K_M.gguf \
  -p "Explain why the sky is blue." -n 128 -st

# server
llama-server -m Qwen3.5-0.8B-abliterated-Q4_K_M.gguf \
  --mmproj mmproj-F16.gguf --port 8080 -c 8192 -ngl 99

With Ollama:

FROM ./Qwen3.5-0.8B-abliterated-Q4_K_M.gguf
PARAMETER num_ctx 8192

mmproj-F16.gguf restores the vision tower. The text quants do not include it, because llama.cpp loads it as a separate projector.

[!WARNING] This model has no refusal behavior. Every quant here inherits that. Quantization does not remove censorship, because there is none left to remove. Do not deploy it in a user-facing product.

---

Caveats

• Perplexity is one number. A 4% perplexity increase can hide larger losses on specific tasks. The chain-of-thought behavior and instruction following were not measured per quant.
• 32 chunks of WikiText-103 is a small sample. Differences under 1% are not meaningful.
• One GPU, one build. Speeds come from a single H200 with a specific llama.cpp commit. Absolute numbers will differ on other hardware.
• The MTP head is present but not useful. llama.cpp can use it with --spec-type draft-mtp, and on CPU that made generation slower in our measurements.

---

Citation

@misc{qwen35-08b-abliterated-gguf,
  title  = {Qwen3.5-0.8B-abliterated GGUF: fifteen quantizations with measured perplexity},
  author = {OpceanAI},
  year   = {2026},
  url    = {https://huggingface.co/OpceanAI/Qwen3.5-0.8B-abliterated-GGUF}
}