Ornith-1.5-35B-A3B-OBLITERATED — low-bit imatrix GGUFs (runs on 12 GB VRAM)

First working IQ2 imatrix quantizations of the abliterated Ornith-1.5-35B-A3B, sized so a 12 GB GPU runs this 35B mixture-of-experts fully on the card. It is an qwen3_5_moe MoE (35B total, only ~3B active per token) — so it is fast — and it keeps vision (image input via the shared mmproj).

Until now the abliterated model only existed as Q4_K_M (19.7 GB) and larger — nothing that fits a mainstream 12 GB card. These are the first IQ2-class GGUFs of it.

!Size vs 12 GB budget

Why these didn't exist

MoE models at 2-bit need a complete importance matrix over all experts or llama-quantize refuses ("the result will be garbage"). This model has 256 experts, top-8 routing — rarely-activated experts are easy to miss. We computed a fresh imatrix on the abliterated weights (bartowski calibration_datav3, -c 512 --parse-special, ~100% executed-tensor coverage, imatrix.gguf ships in this repo) and tuned the per-tensor mix so 2-bit doesn't collapse: ffn_down_exps pinned to iq3_xxs (the sensitive projection), output tensor q6_k, token embeddings q4_k.

Benchmarks (measured, not estimated)

Perplexity & KL-divergence on wikitext-2 (ctx 512), HellaSwag over 400 tasks, against the abliterated Q8_0 as baseline. Speeds via llama-bench on an L40S; on a 12 GB card (RTX 3060 class) tok/s is lower but still high thanks to the 3B active path.

| File | Size | PPL | ΔPPL vs Q8_0 | Mean KLD | Same-top-p | HellaSwag (400) | Target cards |

|---|---|---|---|---|---|---|---|

| IQ2_M | 11.53 GB | 9.92 | +20.5 % | 0.320 | 75.8 % | 77.75 % (−3.0) | 12 GB (best quality, headless / ctx ≤4k with desktop) |

| IQ2_S | 10.88 GB | 10.25 | +24.5 % | 0.372 | 74.3 % | 76.25 % (−4.5) | 12 GB, comfortable + context headroom |

| IQ2_XXS | 9.02 GB | 12.34 | +49.9 % | 0.593 | 68.2 % | 71.75 % (−9.0) | 8–10 GB — usable, clearly degraded |

| Q8_0 (reference) | 34.4 GB | 8.23 | — | — | — | 80.75 % | not in this repo |

!Quality kept at 2-bit

IQ2_M drops just 3 points of HellaSwag vs the full model (77.75 vs 80.75) in 11.5 GB. That is the pick on a 12 GB card. IQ2_S trades a little quality for context/desktop headroom; IQ2_XXS exists so 8–10 GB cards can run it at all (visibly degraded).

How to run

# llama.cpp — text
llama-server -hf cognitivers/Ornith-1.5-35B-A3B-Abliterated-12GB-GGUF:IQ2_M \
  -ngl 999 -c 8192 -fa on -ctk q8_0 -ctv q8_0

# Ollama
ollama run hf.co/cognitivers/Ornith-1.5-35B-A3B-Abliterated-12GB-GGUF:IQ2_M

Vision (image input): download the language file and mmproj-Ornith-1.5-35B-A3B-Abliterated-F16.gguf (~0.9 GB extra VRAM), then:

llama-mtmd-cli -m Ornith-1.5-35B-A3B-Abliterated-IQ2_M.gguf \
  --mmproj mmproj-Ornith-1.5-35B-A3B-Abliterated-F16.gguf \
  --image photo.jpg -p "Describe this image." -ngl 999

Also works in LM Studio, Jan and koboldcpp. The native MTP speculative head is not included (it wasn't in the source GGUF).

Provenance & reproducibility

• Source: Q8_0 from PocketAiHub/Ornith-1.5-35B-A3B-Abliterated-GGUF (Q8_0 ≈ lossless; requantized with --allow-requantize). Original model: ornith-ai/Ornith-1.5-35B-A3B; refusal-direction edit by PocketAI Model Lab.
• Importance matrix: imatrix.gguf in this repo — computed by us on the abliterated weights (llama.cpp, calibration_datav3, -c 512 -b 512 --parse-special).
• Quantized with llama-quantize: base type + --tensor-type ffn_down_exps=iq3_xxs --output-tensor-type q6_k --token-embedding-type q4_k --imatrix imatrix.gguf.

At 2-bit the loss is real (see the table). These exist to make a 35B MoE runnable on mainstream GPUs, not to replace Q4+ if your hardware fits it.

Safety / uncensored

This is an abliterated (uncensored) model: its learned refusals were suppressed, so it will comply with requests an instruct model would decline, and can produce harmful, illegal, or dangerously wrong output more readily. Quantization partially restores some refusals; this is not a safety or truthfulness guarantee. Evaluate and constrain it for your use case.

Credits

• ornith-ai — original Ornith-1.5-35B-A3B.
• PocketAiHub / Pliny — abliteration + reference GGUF conversion.
• bartowski — calibration_datav3.
• llama.cpp — quantization + qwen3_5_moe support.