Austriani // Heretic // GGUF

Qwen3.8-4B-Distill
ARA Abliterated

GGUF Quantizations for llama.cpp 4B Parameters 3/100 Reported Refusals 0.0586 Reported KLD 15/40 Trial Base Architecture This repository hosts GGUF quantizations of Austriani/Qwen3.8-4B-Distill-ARA-Abliterated. This abliterated model inherits the Qwen3.5-4B hybrid decoder: Gated DeltaNet linear-attention plus full-attention layers, with 262,144-token native context. It is derived from empero-ai/Qwen3.8-4B-Distill. Note: Parameter count is reported inconsistently across third-party trackers (4B class vs. ~4.7B). The upstream model card describes it as a 4B weight class; check the upstream config for the exact count. Abliteration Report // ARA Method The base abliteration uses Arbitrary-Rank Ablation (ARA) via Heretic. The optimizer balances three competing objectives: preserve harmless behavior, align harmful outputs with harmless ones, and overcorrect harmful outputs away from their original state. Trial 15 of 40 reported a refusal rate of 3/100 with a KL divergence of 0.0586. These are reported run metrics from the selected trial. Evaluate capability retention on your own tasks before production use. View Full ARA Parameters start_layer_index 3 end_layer_index 21 preserve_good 0.6106 steer_bad 0.0032 overcorrect 0.7700 neighbor_count 12 Available GGUF Quants Note: The large quants use custom layer mixes intended to protect critical attention and boundary layers. Smaller quants use standard llama.cpp formats unless noted. These are GGUF files, but not all recipes are stock quantization presets. 8-Bit & 6-Bit Tier Q8_0 Highest-quality custom mix; critical tensors may be kept at higher precision/F16 where needed. Q6_K High-fidelity K-quant; low expected perplexity loss. 5-Bit Tier Q5_K_M Custom layer mix intended to balance quality and size; critical layers may be bumped in precision. Q5_K_S Smaller custom K-quant targeting high-fidelity performance. 4-Bit Tier Q4_K_M Custom Q4_K_M recipe intended to protect attention and FFN layers. Benchmark against stock Q4_K_M before claiming better performance. Q4_K_S Smaller custom 4-bit recipe focused on memory footprint. Q4_0 Legacy 4-bit format. Faster on some older hardware, but lower quality than K-quants. IQ4_XS Importance-aware 4-bit. Efficient; may outperform standard Q4 formats at a smaller size depending on workload. IQ4_NL Non-linear 4-bit alternative to standard K-quants; performance is hardware-dependent. 3-Bit Tier Q3_K_M Medium 3-bit. Noticeable degradation but retains the core persona. Q3_K_S Small 3-bit. Highly compressed. IQ3_M Medium 3-bit. Highest quality in the IQ3 family. IQ3_S Small 3-bit. Good option for edge devices. IQ3_XS Extra small 3-bit. Slightly larger than XXS, offering better stability. IQ3_XXS Extreme 3-bit. Aggressive size reduction while protecting critical layers. 2-Bit Tier Q2_K Standard 2-bit K-quant. Only recommended if you cannot fit anything larger. IQ2_M Medium 2-bit. Pushes the limits of acceptable quality. IQ2_S Small 2-bit. Quality degradation will be very noticeable. IQ2_XS Extra small 2-bit. Extremely compressed. IQ2_XXS Extreme 2-bit. Bare minimum size. 1-Bit Tier IQ1_M Extreme 1-bit. Highly experimental; the model will likely lose coherence quickly. IQ1_S Smallest possible size. For testing only; not recommended for actual roleplay. Usage & Compatibility These are GGUF files compatible with mainline llama.cpp, LM Studio, KoboldCPP, and Ollama. No custom forks or special builds are required. If you are using ik_llama.cpp and want custom IQK/Trellis formats or the custom MaxOptima mix, please refer to the dedicated ik_llama.cpp repository. Austriani // Qwen3.8-4B-Distill ARA Abliterated // GGUF Base: Austriani/Qwen3.8-4B-Distill-ARA-Abliterated