Austriani // ik_llama.cpp // GGUF
Qwen3.8-4B-Distill
Custom IQK Mix for ik_llama.cpp
4B
Parameters
1024
Imatrix Ctx
500
Cycles
Custom
Quant Mix
Base Architecture
This repository hosts ik_llama.cpp-compatible GGUF quantizations of Austriani/Qwen3.8-4B-Distill-ARA-Abliterated.
The base is derived from empero-ai/Qwen3.8-4B-Distill, a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture. It uses the Qwen3.5 hybrid decoder: Gated DeltaNet linear-attention blocks interleaved with full-attention blocks on every 4th layer (layers 3, 7, 11, 15, 19, 23, 27, 31). Native context is 262,144 tokens.
Note: Parameter count is reported inconsistently across third-party trackers (4B class vs. ~4.7B). The upstream model card describes it as a 4B weight class; check the upstream config for the exact count.
Abliteration Report // ARA Method
The base abliteration uses Arbitrary-Rank Ablation (ARA) via Heretic. The optimizer balances three competing objectives: preserve harmless behavior, align harmful outputs with harmless ones, and overcorrect harmful outputs away from their original state.
Trial 15 of 40 reported a refusal rate of 3/100 with a KL divergence of 0.0586. These are reported run metrics from the selected trial. Evaluate capability retention on your own tasks before production use.
View Full ARA Parameters
start_layer_index
3
end_layer_index
21
preserve_good
0.6106
steer_bad
0.0032
overcorrect
0.7700
neighbor_count
12
Custom Quant Mix
View Full Custom Quant Configuration
^token_embd\.weight=Q6_K,^output\.weight=Q6_K,.*conv1d\.weight=F32,.*ssm_a=F32,.*ssm_dt\.bias=F32,.*\.(attn_norm|post_attention_norm|attn_q_norm|attn_k_norm|ssm_norm|output_norm)\.weight=F32,blk\.0\.ssm_out\.weight=Q8_0,blk\.0\.attn_qkv\.weight=Q6_K,blk\.0\.attn_gate\.weight=Q6_K,blk\.0\.ffn_up\.weight=Q5_K,blk\.0\.ffn_down\.weight=Q5_K,blk\.31\.ffn_up\.weight=Q5_K,blk\.31\.ffn_down\.weight=Q5_K,blk\.(3|7|11|15|19|23|27|31)\.attn_v\.weight=Q6_K,blk\.(3|7|11|15|19|23|27|31)\.attn_output\.weight=Q6_K,blk\.(3|7|11|15|19|23|27|31)\.attn_k\.weight=Q5_K,blk\.(3|7|11|15|19|23|27|31)\.attn_q\.weight=Q5_K,blk\..*\.ssm_out\.weight=IQ5_K,blk\..*\.attn_qkv\.weight=IQ5_K,blk\..*\.attn_gate\.weight=IQ3_KS,blk\..*\.ffn_gate\.weight=IQ3_KS,blk\..*\.ffn_up\.weight=IQ4_KSS,blk\..*\.ffn_down\.weight=IQ4_KSS,.*=Q5_K
Available Quants
Standard IQK // High & Mid Fidelity
IQ5_K
Higher fidelity 5-bit. Best quality-to-size for systems with sufficient memory.
IQ4_K
Standard 4-bit IQK. Uses 256-value super-blocks for scaling.
IQ4_KS
"Small" 4-bit variant with tighter packing. Lower memory footprint.
IQ4_KS_R4
Row-interleaved variant of IQ4_KS, optimized for ik_llama.cpp's matmul kernels.
Trellis // Alternative 4-Bit
IQ4_KT
Dynamic programming trellis quant. Uses a fundamentally different error-minimization strategy than IQK.
Imatrix Specification
This release uses the V3 Calibration Dataset used across previous Austriani releases. The calibration was executed at 500 cycles per chunk with a 1024-token context window, processing approximately 1000 chunks.
Usage & Compatibility
These GGUF files are built exclusively for ik_llama.cpp. IQK and this custom mix will not load in mainline llama.cpp. For mainline llama.cpp / LM Studio / Ollama compatibility, use the standard GGUF repository Austriani/Qwen3.8-4B-Distill-ARA-Abliterated-GGUF instead.
Austriani // Qwen3.8-4B-Distill ARA Abliterated // ik_llama.cpp
Base: Austriani/Qwen3.8-4B-Distill-ARA-Abliterated