Nanbeige-4.3-33B-Heretic GSQ-RCO GGUF

Mixed-precision GSQ-RCO GGUF quantizations of Luni/Nanbeige-4.3-33B-Heretic (an abliterated and de-slopped dense 33B model derived from Nanbeige/Nanbeige4.3-33B). Native support for context lengths up to 524,288 tokens (512K) in llama.cpp.

Why GSQ-RCO?

Standard uniform quantization degrades full-attention reasoning networks at lower bitrates by forcing every tensor into the same bit-width. GSQ-RCO resolves this through two targeted techniques:

1. GSQ (Gumbel-Softmax Quantization): Calibrates per-tensor sensitivity using an in-domain importance matrix (nanbeige_heretic.imatrix, PPL: 26.39) computed across competition mathematics, formal logic, systems engineering algorithms, and adversarial reasoning seeds.
2. RCO (Riemannian Constrained Optimization): Allocates optimal non-uniform precision across the model under a strict global size budget:
   • High-sensitivity attention projections (blk.*.attn_output, attn_v, ffn_down) are promoted to 4-bit (Q4_K).
   • The LM head (output.weight) is preserved at 6-bit (Q6_K) to prevent sampling entropy collapse.
   • Tolerant feedforward tensors (ffn_gate, ffn_up) and attention query/key projections are held at compact 3-bit (IQ3_S).
   • Layer normalization matrices are preserved in unquantized FP32.

Uncensored Heretic Base Model Highlights

• Abliteration Winner: Trial #40 Optuna Pareto selection (-65.4% drop in refusals from baseline) with zero degradation on complex STEM benchmarks.
• Intact Reasoning: Native <think> chain-of-thought circuit remains 100% operational.
• Patched Tokenizer Bug: Fixed upstream duplicate <|im_start|> BOS token emission ("add_bos_token": false).

GGUF Files

Usage

llama.cpp Interactive Chat

llama-cli \
  -m Nanbeige-4.3-33B-Heretic-GSQ-RCO-IQ3_M.gguf \
  -ngl 99 \
  -fa \
  -b 2048 \
  -c 32768 \
  -cnv

llama.cpp Server (OpenAI-Compatible Endpoint)

llama-server \
  -m Nanbeige-4.3-33B-Heretic-GSQ-RCO-IQ3_M.gguf \
  -ngl 99 \
  -fa \
  -b 2048 \
  -c 32768 \
  --port 8080

64k Long-Context on 12 GB GPUs (Low VRAM Mode)

Using 4-bit KV cache (-ctk q4_0 -ctv q4_0) allows 65,536 tokens of context to fit within < 12 GB total VRAM on consumer cards (e.g. RTX 3060 12GB, RTX 4070):

llama-server \
  -m Nanbeige-4.3-33B-Heretic-GSQ-RCO-IQ1_M.gguf \
  -ngl 99 \
  -fa \
  -b 2048 \
  -c 65536 \
  -ctk q4_0 -ctv q4_0 \
  --port 8080

Ollama

ollama run hf.co/Luni/Nanbeige-4.3-33B-Heretic-GSQ-RCO-GGUF:IQ3_M

Attribution & Related Repositories

• Base Model: Nanbeige/Nanbeige4.3-33B by Nanbeige Team (Apache-2.0)
• Uncensored Base BF16: Luni/Nanbeige-4.3-33B-Heretic
• Full GGUF Suite (22 Quant Formats): Luni/Nanbeige-4.3-33B-Heretic-GGUF
• Dedicated EXL3-SC Suite: Luni/Nanbeige-4.3-33B-Heretic-EXL3-SC
• Master EXL3 Collection: Luni/Nanbeige-4.3-33B-Heretic-EXL3
• Stock EXL3 Collection: Luni/Nanbeige-4.3-33B-EXL3