GGUF quantizations of abliterant/Qwen3.5-9B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.5-9B released by Abliterant, for llama.cpp and compatible apps. The repository has 15 quantizations from Q8_0 (9.53 GB) to IQ2_M (3.61 GB) and a BF16 reference. Vision (mmproj-*) and the MTP speculative-decoding head (mtp-*) are separate files.
Quick start · Intended use · What changed · Evaluation · Limitations · Available files
abliterant.ai · Research notes · All formats · Abliterant models
Research model with safety alignment removed. It will produce content the base model refuses. Read Intended use and Limitations before using it. The method, full evaluation and release gates are on the BF16 card.
| Field | Value |
|---|---|
| Developed by | Abliterant |
| Base checkpoint | abliterant/Qwen3.5-9B-RANA-abliterated, derived from Qwen/Qwen3.5-9B |
| Release type | Importance-matrix GGUF quantization of the BF16 reference; not additional training |
| Weight formats | 15 quantizations from Q8_0 to IQ2_M; BF16 reference; separate vision and MTP files |
| Runtime | llama.cpp commit e14a779; tested with CUDA on one RTX 5090 32 GB (Blackwell) |
| Context | 32,768 tokens in the tested example; 262,144 configured, not established for GGUF |
| Closeness to BF16 | KLD 0.0032 (Q8_0), 0.0184 (Q4_K_M), 0.216 (IQ2_M) |
| Compared with a public abliteration | Against bowmanslayer/Qwen3.5-9B-Uncensored: +14.9 pp useful answers to harmful requests (67.4% vs 52.5%), 4.2× smaller capability loss (0.79 vs 3.31 pp) and 2.7× closer to the base model (KL 0.109 vs 0.294); measured on the BF16 build (BF16 card) |
| License | Apache-2.0 |
| Intended use | Refusal-behavior research; see Intended use |
| Format | Repo | Size |
|---|---|---|
| BF16 (reference; Transformers, vLLM) | abliterant/Qwen3.5-9B-RANA-abliterated |
19.3 GB |
| FP8 (block e4m3, Qwen's official FP8 layout; vLLM, SGLang) | abliterant/Qwen3.5-9B-RANA-abliterated-FP8 |
12.2 GB |
| GGUF (llama.cpp; Q8_0 to IQ2_M, plus vision and MTP files) (this repo) | abliterant/Qwen3.5-9B-RANA-abliterated-GGUF |
3.6–9.5 GB per quantization |
| MLX (Apple Silicon; 4-, 5-, 6-, 8-bit and BF16, plus MTP drafter) | abliterant/Qwen3.5-9B-RANA-abliterated-MLX |
6.0–18.8 GB per folder |
| AWQ 4-bit (compressed-tensors; vLLM) | abliterant/Qwen3.5-9B-RANA-abliterated-AWQ |
11.7 GB |
Tested with llama.cpp at commit e14a779 (CUDA) on one RTX 5090 32 GB. Download the model, the vision projector and the MTP head:
hf download abliterant/Qwen3.5-9B-RANA-abliterated-GGUF \
Qwen3.5-9B-RANA-abliterated-Q4_K_M.gguf \
mmproj-Qwen3.5-9B-RANA-abliterated-BF16.gguf \
mtp-Qwen3.5-9B-RANA-abliterated-Q8_0.gguf --local-dir .
Then start the server from the directory that holds them:
llama-server -m Qwen3.5-9B-RANA-abliterated-Q4_K_M.gguf \
--mmproj mmproj-Qwen3.5-9B-RANA-abliterated-BF16.gguf \
-md mtp-Qwen3.5-9B-RANA-abliterated-Q8_0.gguf --spec-type draft-mtp \
--jinja -fa on -ngl 99 -c 32768 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0
| Setting | Value |
|---|---|
| Runtime | llama.cpp at commit e14a779 or later, with llama-server on your PATH |
| Thinking | On by default |
| Sampling | --temp 1.0 --top-p 0.95 --top-k 20 (Qwen's thinking-mode settings); these are also stored as the files' default sampling metadata |
min_p |
Set --min-p 0 explicitly; llama.cpp's own default min_p is 0.05 |
Context (-c) |
32,768 tokens in the example; raise it for long coding or technical answers |
| MTP speculative decoding | -md mtp-…-Q8_0.gguf --spec-type draft-mtp |
| Vision | --mmproj mmproj-…-BF16.gguf (or the F16 file) |
Q8_0 and Q6_K are near-lossless. Q5_K_M and Q4_K_M are the usual choices when memory is tight. Below 4 bits the KLD rises quickly; Q3 and IQ3 files fit about 4–5 GB with a visible quality cost, and Q2_K and IQ2_M are the smallest files with the largest drift. Per-file size and KLD are in the table under Available files; functional checks were run on Q4_K_M only.
RANA (Reasoning-Anchored, Norm-preserving Ablation) is Abliterant's refusal-ablation method for reasoning models: it anchors the edit to the model's own reasoning, keeps the edited weights at their original norms, and keeps the reference model's vision, tool calling and speculative decoding working. Its design, full BF16 evaluation and release gates are documented on the BF16 card. This conversion is not additional training.
convert_hf_to_gguf.py (commit e14a779). Tensor counts, dimensions and types match unsloth's and lmstudio-community's conversions of the base model (table below).llama-imatrix on the BF16 GGUF over 601 chunks of 512 tokens of bartowski's public calibration text (Qwen3.8-27B-calibration-v6.txt from bartowski/Qwen3.8-27B-GGUF). General text, no refusal-related prompts.llama-quantize --imatrix with llama.cpp's default tensor layouts for each type (no per-tensor overrides). The MTP head is quantized to Q8_0 without an importance matrix.| Component | Converter flag | Tensors | Matches |
|---|---|---|---|
| Main model | --no-mtp |
427 | unsloth Qwen3.5-9B-BF16.gguf: all 427 tensors |
| MTP head | --mtp |
18 | the MTP layer with the shared token embeddings and output head |
| Vision tower | --mmproj |
334 | lmstudio-community and unsloth mmproj files |
All 15 quantizations were measured for closeness to the BF16 GGUF; Q4_K_M was served for functional checks. The harmful-request, capability and benign-task evaluations of the BF16 card were not repeated on this format.
llama-perplexity --kl-divergence on wiki.test.raw, 100 chunks of 512 tokens, the protocol of the release family's 27B GGUF card.mmproj-…-BF16 and mtp-…-Q8_0, served by llama-server with the command in Quick start.e14a779, CUDA, one RTX 5090 32 GB.results/.The BF16 GGUF has a perplexity of 8.052 ± 0.129 on the same text.
| Quantization | Size | KLD | Same top token | Perplexity |
|---|---|---|---|---|
| Q8_0 | 9.53 GB | 0.0032 | 98.4% | 8.056 |
| Q6_K | 7.36 GB | 0.0058 | 97.3% | 8.059 |
| Q5_K_M | 6.47 GB | 0.0077 | 96.6% | 8.063 |
| Q5_K_S | 6.31 GB | 0.0077 | 96.3% | 8.056 |
| Q4_K_M | 5.63 GB | 0.0184 | 93.9% | 8.126 |
| Q4_K_S | 5.35 GB | 0.0205 | 93.6% | 8.126 |
| IQ4_XS | 5.20 GB | 0.0256 | 92.9% | 8.159 |
| IQ4_NL | 5.42 GB | 0.0254 | 92.7% | 8.173 |
| Q4_0 | 5.33 GB | 0.0519 | 90.0% | 8.468 |
| Q3_K_L | 4.93 GB | 0.0754 | 88.0% | 8.527 |
| Q3_K_M | 4.62 GB | 0.0769 | 87.7% | 8.485 |
| IQ3_M | 4.42 GB | 0.0793 | 87.5% | 8.579 |
| IQ3_XXS | 3.94 GB | 0.1178 | 84.4% | 8.759 |
| Q2_K | 3.83 GB | 0.1950 | 80.1% | 9.198 |
| IQ2_M | 3.61 GB | 0.2160 | 79.3% | 9.554 |
Served with llama-server, Q4_K_M + mmproj-…-BF16 + mtp-…-Q8_0:
--min-p 0 explicitly, because llama.cpp's own default min_p is 0.05.Other formats are listed under Release family.
KLD is the mean KL divergence of each quantization's next-token distribution from the BF16 GGUF; "same top token" is how often both pick the same most likely token (llama-perplexity on wiki.test.raw, 100 chunks of 512 tokens).
| File | Size | KLD | 99th pct KLD | Same top token | Notes |
|---|---|---|---|---|---|
Qwen3.5-9B-RANA-abliterated-Q8_0.gguf |
9.53 GB | 0.0032 | 0.009 | 98.4% | near-lossless |
Qwen3.5-9B-RANA-abliterated-Q6_K.gguf |
7.36 GB | 0.0058 | 0.022 | 97.3% | near-lossless |
Qwen3.5-9B-RANA-abliterated-Q5_K_M.gguf |
6.47 GB | 0.0077 | 0.042 | 96.6% | |
Qwen3.5-9B-RANA-abliterated-Q5_K_S.gguf |
6.31 GB | 0.0077 | 0.050 | 96.3% | |
Qwen3.5-9B-RANA-abliterated-Q4_K_M.gguf |
5.63 GB | 0.0184 | 0.130 | 93.9% | functional checks run on this file |
Qwen3.5-9B-RANA-abliterated-Q4_K_S.gguf |
5.35 GB | 0.0205 | 0.146 | 93.6% | |
Qwen3.5-9B-RANA-abliterated-IQ4_XS.gguf |
5.20 GB | 0.0256 | 0.198 | 92.9% | |
Qwen3.5-9B-RANA-abliterated-IQ4_NL.gguf |
5.42 GB | 0.0254 | 0.203 | 92.7% | |
Qwen3.5-9B-RANA-abliterated-Q4_0.gguf |
5.33 GB | 0.0519 | 0.420 | 90.0% | |
Qwen3.5-9B-RANA-abliterated-Q3_K_L.gguf |
4.93 GB | 0.0754 | 0.623 | 88.0% | |
Qwen3.5-9B-RANA-abliterated-Q3_K_M.gguf |
4.62 GB | 0.0769 | 0.631 | 87.7% | |
Qwen3.5-9B-RANA-abliterated-IQ3_M.gguf |
4.42 GB | 0.0793 | 0.585 | 87.5% | |
Qwen3.5-9B-RANA-abliterated-IQ3_XXS.gguf |
3.94 GB | 0.1178 | 0.941 | 84.4% | |
Qwen3.5-9B-RANA-abliterated-Q2_K.gguf |
3.83 GB | 0.1950 | 1.529 | 80.1% | |
Qwen3.5-9B-RANA-abliterated-IQ2_M.gguf |
3.61 GB | 0.2160 | 1.666 | 79.3% |
Also in this repository:
Qwen3.5-9B-RANA-abliterated-BF16.gguf (17.92 GB): the unquantized GGUF, the KLD reference.mmproj-Qwen3.5-9B-RANA-abliterated-{BF16,F16}.gguf (0.92 GB each): the vision tower, identical to the base model's.mtp-Qwen3.5-9B-RANA-abliterated-{BF16,Q8_0}.gguf: the MTP head as a speculative draft for --spec-type draft-mtp, consistent with the main model.Qwen3.5-9B-RANA-abliterated-imatrix.gguf: the importance matrix used for every quantization.LICENSE: the Apache-2.0 license inherited from Qwen.assets/abliterant-model-card-header.png: the card banner.results/: kld.json (closeness per quantization), ppl_bf16.json, smoke_Q4_K_M.json (functional checks), verify_gguf.json (conversion checks).e14a779.results/.Qwen/Qwen3.5-9B.wiki.test.raw).Have the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo abliterant/Qwen3.5-9B-RANA-abliterated-GGUF ./model-folderabliterant/Qwen3.5-9B-RANA-abliterated