Qwen3.8-27B-RANA-abliterated-GGUF

GGUF quantizations of abliterant/Qwen3.8-27B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.8-27B released by Abliterant, for llama.cpp and compatible apps. The repository has 15 quantizations from Q8_0 (28.60 GB) to IQ2_M (10.00 GB) and a split BF16 reference. Vision (mmproj-*) and the MTP speculative-decoding head (mtp-*) are included as separate files, in the same layout as ggml-org/Qwen3.8-27B-GGUF.

Quick start · Intended use · What changed · Evaluation · Limitations · Available files

Try RANA in Abliterant Chat · abliterant.ai · Research notes · All formats · Abliterant models

Research model with safety alignment removed. It will produce content the base model refuses. Read Intended use and Limitations before using it. The method, full evaluation and release gates are on the BF16 card.

At a glance

Release family

Quick start

The release tests used llama.cpp at commit 4b1a27f (CUDA, RTX PRO 6000 Blackwell). Use a build of that revision with llama-server on your PATH.

To run from local files, first download the model, the vision projector and the MTP head:

hf download abliterant/Qwen3.8-27B-RANA-abliterated-GGUF \
  Qwen3.8-27B-RANA-abliterated-Q4_K_M.gguf \
  mmproj-Qwen3.8-27B-RANA-abliterated-BF16.gguf \
  mtp-Qwen3.8-27B-RANA-abliterated-Q8_0.gguf --local-dir .

Then start the server from the directory that holds them:

llama-server -m Qwen3.8-27B-RANA-abliterated-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-RANA-abliterated-BF16.gguf \
  -md mtp-Qwen3.8-27B-RANA-abliterated-Q8_0.gguf --spec-type draft-mtp \
  --jinja -fa on -ngl 99 -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0

Alternatively, llama.cpp can fetch the files itself; the release also tested this path. It picks up the mmproj-* file automatically, but the mtp-* file only when --spec-type draft-mtp is given.

llama-server -hf abliterant/Qwen3.8-27B-RANA-abliterated-GGUF:Q4_K_M --spec-type draft-mtp \
  --jinja -fa on -ngl 99 -c 32768 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0

Recommended settings

• Long technical requests can need 20–50k tokens of reasoning (see the FP8 card). Raise -c to match; the native context is 262k.

Choosing a quantization

Which one? Q8_0 and Q6_K are near-lossless. Q5_K_M and Q4_K_M are the usual choices when memory is tight. Below 4 bits the KLD rises quickly. IQ3/Q3 and IQ2/Q2 are for fitting into 12–16 GB, with a visible quality cost.

Per-file size and KLD are listed in the table under Available files; functional and refusal checks were run on Q4_K_M only.

Intended use

• Research only: interpretability, red-teaming, and robustness evaluation of refusal behavior.
• Responsibility: You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen, and the terms of any platform where outputs are used.

Out of scope

• Not for public or end-user deployment without a separate moderation layer. The model's own refusals have been largely removed, so any safety filtering has to happen outside it.

What changed

The refusal-abliteration method, full BF16 evaluation and release gates are documented on the BF16 release card. This GGUF conversion and quantization are not additional training.

• Source: the published BF16 repo, converted with llama.cpp's convert_hf_to_gguf.py (commit 4b1a27f). Tensor counts and sizes match ggml-org's conversion of the base model (table below).
• Importance matrix (imatrix): activation statistics that GGUF quantization uses to decide which weights need more precision, computed with llama-imatrix on the BF16 GGUF with bartowski's public calibration text (Qwen3.8-27B-calibration-v6.txt from bartowski/Qwen3.8-27B-GGUF). General text, no refusal-related prompts.
• Quantization: llama-quantize --imatrix with llama.cpp's default tensor layouts for each type (no per-tensor overrides).

Evaluation

Functional and refusal checks cover Q4_K_M only. All 15 quantizations have measurements of how closely they track BF16, which is not a full downstream evaluation. Closeness is measured as KLD (Kullback–Leibler divergence: how far a quant's next-token probabilities have moved from the BF16 GGUF's; 0 means identical) on 100 chunks of 512 tokens of wiki.test.raw.

Setup

• Builds: all 15 quantizations against the BF16 GGUF for closeness; Q4_K_M for functional and refusal checks, with the BF16 and FP8 builds as the refusal comparison.
• Closeness protocol: KLD with llama-perplexity on wiki.test.raw, 100 chunks of 512 tokens, against the BF16 GGUF; bartowski's quants of the base model (his perplexity.md, llama.cpp b10896) as an indicative reference.
• Functional checks: Q4_K_M served with llama-server for vision, tool calling and MTP speculative decoding; repeated with the files fetched via -hf … --spec-type draft-mtp.
• Refusal suite: refusal suite v2 on HarmBench (200) and a held-out set (240), seed 1, thinking on, 16k-token budget, judged by openai/gpt-oss-safeguard-20b.
• Engines and hardware: the GGUF on llama.cpp commit 4b1a27f (CUDA, RTX PRO 6000 Blackwell); the BF16 and FP8 builds on vLLM.
• Raw results: results/ in this repository.

Quantization quality

The table below puts each quant next to bartowski's quant of the base model, measured with the same protocol (his perplexity.md, llama.cpp b10896). It is a different model, so the comparison is indicative only.

• The BF16 GGUF has a perplexity of 6.776 ± 0.104 on the same text, against 6.744 ± 0.103 for base Qwen3.8-27B in bartowski's table.
• Most of his files are larger at the same name, by up to 2.3 GB, because he overrides the type of some tensors; these quants use llama.cpp's default layouts.
• At equal size the two are close: this repo's Q4_K_M (16.55 GB) has the same KLD as his Q4_K_S (16.36 GB), 0.0156, and this IQ4_XS has a slightly lower KLD than his at 0.4 GB smaller.

Functional checks (Q4_K_M, llama-server)

Served with llama-server, Q4_K_M + mmproj-…-BF16 + mtp-…-Q8_0:

• Vision: reads the code word and shape from a synthetic image: pass.
• Tool calling: 3-turn call → result → second call with a new argument: pass.
• MTP speculative decoding: 348 of 486 drafted tokens accepted (71.6%) on one 510-token generation at temperature 0: pass.
• The same three checks pass when the files are fetched with -hf … --spec-type draft-mtp.

Refusals (Q4_K_M, seed 1)

Same prompts, seed and sampling as the BF16 and FP8 builds (refusal suite v2, seed 1, thinking on, 16k-token budget), judged by openai/gpt-oss-safeguard-20b. The GGUF was served with llama.cpp, the other two with vLLM, so part of any difference can come from the engine. In the table, "hard" and "soft" count refusals; a budget hit is a response that reaches the 16k-token limit before finishing.

• HarmBench labels follow the rule used for every build: each gpt-oss HARD_REFUSAL is re-judged (BF16 card, G1 re-adjudication disclosure). Q4_K_M and BF16 had none in seed 1; FP8's 3 raw ones were re-judged as 2 answers and 1 degenerate, as shown above.
• Paired with BF16 on the same prompts (exact McNemar test, which compares two models on the prompts where only one of them did something): answers on HarmBench 13 Q4_K_M-only vs 11 BF16-only (p = 0.84), held-out 10 vs 13 (p = 0.68); budget hits on HarmBench 5 vs 11 (p = 0.21), held-out 4 vs 7 (p = 0.55). None of the differences is significant.
• The one held-out hard refusal is a raw label on a finished answer that contains no refusal phrase; it was not re-judged.

Limitations

• Functional checks and refusal behavior were evaluated on Q4_K_M only, one seed and one judge. Other quants were checked for closeness to BF16 (KLD) but not for refusal behavior; the lowest-bit quants drift the most from BF16 and may behave differently.
• KLD is measured on English Wikipedia text at 512-token context. It says how closely a quant tracks BF16, not how it scores on downstream tasks.
• Everything listed under Limitations on the BF16 card applies here too: judge-measured refusal rates, long reasoning on technical requests, and the capability changes measured there.

Reduced refusal does not establish greater safety, accuracy, factual reliability, or universal compliance. Because the GGUF ran on llama.cpp while BF16 and FP8 ran on vLLM, not every difference in the comparison can be attributed to quantization alone.

Recommendations

• Context: raise -c for long technical requests, which can need 20–50k tokens of reasoning; see Recommended settings.
• Sampling: set --min-p 0 explicitly, because llama.cpp's own default min_p is 0.05.
• Safety filtering: keep it in a separate moderation layer outside the model; see Out of scope.
• Quantization choice: Q8_0 and Q6_K are near-lossless; Q5_K_M and Q4_K_M are the usual choices when memory is tight; see Choosing a quantization.

Available files

KLD is the mean KL divergence of each quant's next-token distribution from the BF16 GGUF. "Same top token" is how often both pick the same most likely token. Measured with llama-perplexity on wiki.test.raw, 100 chunks of 512 tokens. Lower KLD is closer to BF16.

Also in this repo:

• mmproj-Qwen3.8-27B-RANA-abliterated-{BF16,F16}.gguf (0.93 GB): the vision tower. Same weights as the base model's (abliteration does not touch it).
• mtp-Qwen3.8-27B-RANA-abliterated-{Q8_0,BF16}.gguf (3.2 / 5.9 GB): the MTP head as a speculative draft for --spec-type draft-mtp. It is the abliterated MTP head, consistent with the main model.
• Qwen3.8-27B-RANA-abliterated-BF16/ (2 parts, 53.8 GB): unquantized GGUF, the KLD reference.
• Qwen3.8-27B-RANA-abliterated-imatrix.gguf: the importance matrix used for every quant.
• SHA256SUMS, results/: checksums and the measurements behind this card.

Provenance and license

The artifacts retain the Apache-2.0 license inherited from Qwen; see the Apache-2.0 terms and the upstream model. Original release preparation and measurements are credited to preemware, alongside the contributors below.

Credits

• Qwen team: base model Qwen/Qwen3.8-27B.
• Arditi et al., 2024: "Refusal in Language Models Is Mediated by a Single Direction".
• Jim Lai (grimjim): prior work on norm-preserving abliteration.
• llama.cpp / ggml-org: conversion, quantization and inference; bartowski: the calibration text and the public KLD table used for comparison.
• Benchmarks/datasets: HarmBench, StrongREJECT, JailbreakBench, CategoricalHarmfulQA; wikitext-2.