Austriani // ik_llama.cpp // GGUF

Qwen3.8-4B-Distill
ARA Abliterated

Custom IQK Mix for ik_llama.cpp 4B Parameters 1024 Imatrix Ctx 500 Cycles Custom Quant Mix Base Architecture This repository hosts ik_llama.cpp-compatible GGUF quantizations of Austriani/Qwen3.8-4B-Distill-ARA-Abliterated. The base is derived from empero-ai/Qwen3.8-4B-Distill, a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture. It uses the Qwen3.5 hybrid decoder: Gated DeltaNet linear-attention blocks interleaved with full-attention blocks on every 4th layer (layers 3, 7, 11, 15, 19, 23, 27, 31). Native context is 262,144 tokens. Note: Parameter count is reported inconsistently across third-party trackers (4B class vs. ~4.7B). The upstream model card describes it as a 4B weight class; check the upstream config for the exact count. Abliteration Report // ARA Method The base abliteration uses Arbitrary-Rank Ablation (ARA) via Heretic. The optimizer balances three competing objectives: preserve harmless behavior, align harmful outputs with harmless ones, and overcorrect harmful outputs away from their original state. Trial 15 of 40 reported a refusal rate of 3/100 with a KL divergence of 0.0586. These are reported run metrics from the selected trial. Evaluate capability retention on your own tasks before production use. View Full ARA Parameters start_layer_index 3 end_layer_index 21 preserve_good 0.6106 steer_bad 0.0032 overcorrect 0.7700 neighbor_count 12 Custom Quant Mix View Full Custom Quant Configuration ^token_embd\.weight=Q6_K,^output\.weight=Q6_K,.*conv1d\.weight=F32,.*ssm_a=F32,.*ssm_dt\.bias=F32,.*\.(attn_norm|post_attention_norm|attn_q_norm|attn_k_norm|ssm_norm|output_norm)\.weight=F32,blk\.0\.ssm_out\.weight=Q8_0,blk\.0\.attn_qkv\.weight=Q6_K,blk\.0\.attn_gate\.weight=Q6_K,blk\.0\.ffn_up\.weight=Q5_K,blk\.0\.ffn_down\.weight=Q5_K,blk\.31\.ffn_up\.weight=Q5_K,blk\.31\.ffn_down\.weight=Q5_K,blk\.(3|7|11|15|19|23|27|31)\.attn_v\.weight=Q6_K,blk\.(3|7|11|15|19|23|27|31)\.attn_output\.weight=Q6_K,blk\.(3|7|11|15|19|23|27|31)\.attn_k\.weight=Q5_K,blk\.(3|7|11|15|19|23|27|31)\.attn_q\.weight=Q5_K,blk\..*\.ssm_out\.weight=IQ5_K,blk\..*\.attn_qkv\.weight=IQ5_K,blk\..*\.attn_gate\.weight=IQ3_KS,blk\..*\.ffn_gate\.weight=IQ3_KS,blk\..*\.ffn_up\.weight=IQ4_KSS,blk\..*\.ffn_down\.weight=IQ4_KSS,.*=Q5_K Available Quants Standard IQK // High & Mid Fidelity IQ5_K Higher fidelity 5-bit. Best quality-to-size for systems with sufficient memory. IQ4_K Standard 4-bit IQK. Uses 256-value super-blocks for scaling. IQ4_KS "Small" 4-bit variant with tighter packing. Lower memory footprint. IQ4_KS_R4 Row-interleaved variant of IQ4_KS, optimized for ik_llama.cpp's matmul kernels. Trellis // Alternative 4-Bit IQ4_KT Dynamic programming trellis quant. Uses a fundamentally different error-minimization strategy than IQK. Imatrix Specification This release uses the V3 Calibration Dataset used across previous Austriani releases. The calibration was executed at 500 cycles per chunk with a 1024-token context window, processing approximately 1000 chunks. Usage & Compatibility These GGUF files are built exclusively for ik_llama.cpp. IQK and this custom mix will not load in mainline llama.cpp. For mainline llama.cpp / LM Studio / Ollama compatibility, use the standard GGUF repository Austriani/Qwen3.8-4B-Distill-ARA-Abliterated-GGUF instead. Austriani // Qwen3.8-4B-Distill ARA Abliterated // ik_llama.cpp Base: Austriani/Qwen3.8-4B-Distill-ARA-Abliterated