Nanbeige4.3-33B-RVN-Uncensored-Heretic-Abliterated-RCO-GGUF

A refusal-ablated Nanbeige/Nanbeige4.3-33B, as GGUF for 16–24 GB GPUs, led by an 11.6 GB RCO mixed-precision build that runs on a 16 GB GPU with a 16K context.

This repo holds the GGUFs. The BF16 weights are in Nanbeige4.3-33B-RVN-Heretic-Abliterated-Uncensored. Every number on this card was measured by us, and the method below is described in enough detail to reproduce it.

Which file do I want?

llama-server -m Nanbeige4.3-33B-RVN-Heretic-RCO-2.60bpw-IQ2_S.gguf -ngl 99 -c 16384 \
  -ctk q8_0 -ctv q8_0 -fa on --jinja

VRAM measured on one GPU with -c 16384 -ctk q8_0 -ctv q8_0 -fa on after a warm-up request: RCO 2.60 13,081 MiB (about 3 GB free on 16 GB), IQ3_S 16,223 MiB (too tight for a 16 GB card at 16K, fine on 20 GB), IQ4_XS 19,191 MiB (about 5 GB free on 24 GB). Decoding runs at about 100 tok/s on an RTX PRO 6000.

Please read Known issue: unclosed thinking before you deploy it. On some prompts the answer lands inside the reasoning block.

---

Results at a glance

• IQ3_S and IQ4_XS are standard llama.cpp mixes made with this model's own imatrix (embedding and output stay at higher precision).
• PPL error is about ±0.25 and WinoGrande about ±2.1, so IQ3_S, IQ4_XS and Q4_K_M are statistically the same quality. Q4_K_M is not shipped: it is 2 GB larger than IQ4_XS for no measurable gain, and needs 21,123 MiB at 16K, which is tight on a 24 GB card.

---

Plain llama.cpp quants (for comparison or if you prefer a single standard type)

Standard llama.cpp mixes made with the same imatrix (embedding and output stay at higher precision). Same measurement setup as above.

At a similar size, the RCO 2.60 build (11.6 GB, PPL 25.51) loses much less than plain IQ2_S or IQ2_XS. IQ3_XXS fits a 16 GB card at 16K but leaves only about 1.5 GB free.

---

How we chose the method

The brief was: remove refusals with minimal damage, then fit a 33B on a 16 GB card with a real KV cache. We worked through it in that order.

Step 1: abliteration with heretic ARA (not plain abliteration)

• Tool: heretic, ara branch, commit edc3b12, full-weight ARA (--use-ara --no-use-ara-lora). No quantization was used during the search.
• Why ARA: it optimizes refusal removal and KL against the original jointly, instead of projecting out one direction and hoping. Our rule was "low refusals, but minimum damage first".
• Search: two independent Optuna studies of 100 trials each, one per GPU.
• Two details that matter for this model:
  1. System prompt. Nanbeige's chat template injects its own default system prompt. We used that prompt for the direction search, scoring and KL, not heretic's generic "You are a helpful assistant.".
  2. Thinking. The model refuses after the reasoning block. A 100-token search window only sees the reasoning, so trials forced a response prefix that jumps to the answer. Final acceptance used a long eval that scores only the post-thinking text.
• Released: trial 56. Its parameters are in heretic_trial.json.
• Rejected: trial 48. Its KL was lower (0.0068), but it left 24/100 harmful prompts unfinished, against 12 for trial 56, and it refused one harmless prompt.

Prompts were mlabonne/harmful_behaviors and mlabonne/harmless_alpaca, first 100 of each test split, with greedy decoding. Other abliterated cards use different prompts and scoring, so their numbers are not directly comparable.

Step 2: the quantization method, and why it is not GSQ

We considered two routes:

Why we did not run GSQ:

1. The target is llama.cpp, and GSQ's gain does not survive the trip into GGUF. GSQ optimizes weights for its own quantization grid. llama.cpp's 2–3 bit types (IQ2_XS, IQ2_S, IQ3_XXS) are different codebook formats. Turning a GSQ checkpoint into a GGUF means quantizing it again into those formats, and that second step throws away the rounding GSQ learned. The 16 GB user would get none of the benefit.
2. RCO alone attacks the part that matters at 2.6 bpw. At this budget the biggest lever is where the bits go: which tensors get 3 bits and which get 2. RCO optimizes exactly that, against the model's real output distribution, using the real llama.cpp bit patterns.

What this means for the file: the released GGUF is an RCO type allocation over standard llama.cpp quantizers. No GSQ weight refinement was applied. True "GSQ inside GGUF", meaning learned rounding within the IQ2/IQ3 codebooks, would be a research project, and it is open work.

Step 3: the RCO pipeline (what we built)

1. imatrix. Computed with llama-imatrix on this model's own F16 GGUF, 334 chunks × 2048 tokens.
2. Candidates. The whole model is quantized once per candidate type with llama-quantize --pure --imatrix: IQ2_XS (2.31 bpw), IQ2_S (2.56) and IQ3_XXS (3.06). These are real llama.cpp bit patterns, not simulated quantization.
3. GGUF → RCO adapter (ours). Each candidate is dequantized tensor by tensor into RCO's per-layer weight database, undoing llama.cpp's q/k RoPE permutation. On the F16 GGUF the round trip matches the HF checkpoint to max |Δ| = 3e-8.
4. RCO search. RCO's optimizer core runs unmodified (commit 9a1e09c): projected Gumbel-softmax on the budget manifold, finished with exact DP rounding.
   • Objective: KL(original ‖ quantized) on the full output, with the budget held at exactly 2.60 bpw.
   • Settings: 64 × 2048 fineweb-edu tokens, 200 steps.
   • We made two memory-only changes so the search fits on 2× 96 GB:
     • candidate deltas are stored in per-row FP8 and recomputed in backward;
     • gradient checkpointing is forced on, because HF silently skips it in eval mode.
5. Materialize. The allocation becomes a --tensor-type-file and goes through llama-quantize --imatrix. Embedding is Q4_K, output is Q6_K.
6. Verify. We read the GGUF header back and compared it to the map: 0 of 364 tensors mismatched. Then we measured PPL, VRAM, HellaSwag, WinoGrande and chat behaviour.

What the optimizer chose: 186 × IQ2_XS, 33 × IQ2_S, 145 × IQ3_XXS. It spent bits on attn_v (32/52 at 3 bit), attn_k and attn_output, and saved them on attn_q and the FFN. The full map is in tensor-allocation/.

---

Evidence

All numbers come from one machine (2× RTX PRO 6000 Blackwell 96 GB) running llama.cpp commit 1b4554e.

Perplexity (wikitext-2-raw test, -c 4096)

Thinking models have high absolute wikitext PPL, so compare the deltas. Pure IQ3_XXS scores badly because --pure also pushes the embedding and output layers to 3 bits.

Zero-shot tasks (llama-perplexity --hellaswag / --winogrande)

• On WinoGrande, RCO beats the same-size IQ2_S by 4.6 points, about twice the standard error.
• With 400 HellaSwag tasks the 95 % interval is about ±4.5 points, so the HellaSwag gaps are within noise.

Negative result: a longer search did not help

• Lower calibration KL did not carry over to held-out PPL.
• Wikitext PPL was used both to choose among these runs and as the reported number, so treat 25.51 as a selected result.
• What survives that caveat: RCO allocation beat every single-type build of this model at its size, on PPL and on WinoGrande. Multi-seed searches and a held-out KL are open work.

---

Known issue: unclosed thinking

On some prompts the model writes its whole answer but never emits the closing </think> tag. A reasoning-aware server (llama-server's default) then files everything under reasoning_content, and the visible content comes back empty. The answer is there, just in the wrong field.

We measured this on 100 harmless prompts (the harmless eval set) with each model's chat template, temperature 0.6, a fixed per-prompt seed, max 4000 tokens, 16K context and q8_0 KV:

What this shows:

• The behaviour already exists in the original model (8/100).
• Abliteration changes it little (11/100, within the noise of 100 prompts).
• 2.6-bit quantization roughly doubles it (19–20/100). This comes from the bit width, not from RCO: the same-size pure IQ2_S does the same.
• Counting both fields, answer quality is close across the board (94–99/100).

Workarounds:

• Run the server with --reasoning-format none and split on </think> in your client. If no closing tag appears, treat the whole text after <think> as the answer.
• Or keep the default parsing, and when content is empty but reasoning_content is not, show reasoning_content.
• Give it a generous token budget (-n 2048 or more), since it thinks first.

---

Files

Reproduce the released GGUF

llama-quantize --imatrix imatrix-Nanbeige4.3-33B-RVN-Heretic.gguf \
  --tensor-type-file tensor-allocation/RCO-2.60bpw.tensor_types.txt \
  --token-embedding-type q4_k --output-tensor-type q6_k \
  Nanbeige4.3-33B-RVN-Heretic-F16.gguf out.gguf IQ2_S

To redo the whole search:

1. Build the pure candidates with llama-quantize --pure --imatrix (under --pure, the IQ2_S ftype writes IQ2_XS tensors; use IQ2_M to get IQ2_S tensors).
2. Dequantize each candidate tensor by tensor into RCO's per-layer weight database, undoing llama.cpp's q/k RoPE permutation.
3. Run RCO's search: KL objective, 2.60 bpw budget, 64 × 2048 fineweb-edu tokens, 200 steps.
4. Write the allocation as a --tensor-type-file, quantize with the command above, and check the GGUF header against the map.

Usage (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
m = "0bserverx/Nanbeige4.3-33B-RVN-Heretic-Abliterated-Uncensored"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "Explain how a TLS handshake works."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=2048, use_cache=True)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

The upstream config.json sets "use_cache": false, so pass use_cache=True or decoding is very slow.

Limitations & Responsible Use

• Reduced safety guardrails by design. This model is not intended for use in applications requiring robust safety filtering, content moderation, or deployment to minors.
• Certain guardrails are intentionally left in place. Abliteration targets refusal behavior on general harmful-prompt categories; a small set of hard safety-trained categories is deliberately not fully removed. Behavior may vary across domains and languages.
• Beyond PPL, HellaSwag, WinoGrande and the refusal eval, no capability benchmarks (MMLU and similar) have been run.
• Not affiliated with or endorsed by Nanbeige.

Research use and responsibility

This repository is intended for legitimate research and controlled evaluation, including interpretability, alignment and refusal-behavior analysis, red-team testing, and robustness work. It is not a ready-made production safety layer. If you deploy the model or expose it to other users, you are responsible for adding suitable access controls, moderation, monitoring, and abuse prevention.

Use of these files is subject to the Apache License 2.0 and all applicable laws. You are responsible for how you operate the model and for outputs produced in your environment. To the extent permitted by law, the repository maintainers and upstream authors accept no liability for misuse or resulting harm. Generated outputs are not statements or endorsements by the maintainers, upstream creators, or their organizations.

License

Apache-2.0, inherited from Nanbeige4.3-33B.

Credits

Nanbeige (base model) · heretic (ARA) · IST-DASLab/RCO (bit allocation) · llama.cpp (quantizers, runtime, benchmarks)