What's that?
The goal: Make a quantization of ApocalypseParty/Qwen3.5-27B-v3-SFT-2 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead.
The idea: I make quants that I run on my machine with my constraints. For smaller dense models, that means 32k context, full VRAM offload with KV resident in VRAM, running ik_llama.cpp specifically. That's what this one is designed to do.
quantized with ik_llama.cpp build: c069e89e from bf16, using imatrix generated on bartowski's combined dataset on q8_0 weights with -c 512 -b 512 -ub 512
This repo provides the imatrix in .gguf and .dat, the q8_0 GGUF, and the mixed precision 16G-oriented GGUF described below
Incompatible with mainline llama.cpp. It is quantized specifically to be used with ik_llama.cpp and its derivatives. It utilizes quantization types not supported by llama.cpp as of 10/07/2026.
Layout
| Layer | Dims | Dims | Quant |
|---|---|---|---|
| token_embd | 5120 | 248320 | iq5_ks |
| DeltaNet blk | 48 | ||
| attn_gate | 5120 | 6144 | iq5_ks |
| attn_norm | 5120 | 1 | f32 |
| attn_qkv | 5120 | 10240 | iq5_ks |
| ffn_down | 17408 | 5120 | iq4_kt |
| ffn_gate | 5120 | 17408 | iq3_kt |
| ffn_norm | 5120 | 1 | f32 |
| ffn_up | 5120 | 17408 | iq3_kt |
| post_attention_norm | 5120 | 1 | f32 |
| ssm_a | 48 | 1 | f32 |
| ssm_alpha | 5120 | 48 | f32 |
| ssm_beta | 5120 | 48 | bf16 |
| ssm_conv1d | 4 | 10240 | f32 |
| ssm_dt.bias | 48 | 1 | f32 |
| ssm_norm | 128 | 1 | f32 |
| ssm_out | 6144 | 5120 | iq4_kt |
| Full attn blk | 16 | ||
| attn_k | 5120 | 1024 | iq6_k |
| attn_k_norm | 256 | 1 | f32 |
| attn_norm | 5120 | 1 | f32 |
| attn_q | 5120 | 12288 | iq5_ks |
| attn_q_norm | 256 | 1 | f32 |
| attn_v | 5120 | 1024 | iq6_k |
| ffn_down | 17408 | 5120 | iq4_kt |
| ffn_gate | 5120 | 17408 | iq3_kt |
| ffn_up | 5120 | 17408 | iq3_kt |
| post_attention_norm | 5120 | 1 | f32 |
| attn_output | 6144 | 5120 | iq5_ks |
| blk64 | 1 | ||
| attn_k | 5120 | 1024 | iq6_k |
| attn_k_norm | 256 | 1 | f32 |
| attn_norm | 5120 | 1 | f32 |
| attn_q | 5120 | 12288 | iq6_k |
| attn_q_norm | 256 | 1 | f32 |
| attn_v | 5120 | 1024 | iq6_k |
| ffn_down | 17408 | 5120 | iq4_kt |
| ffn_gate | 5120 | 17408 | iq4_kt |
| ffn_up | 5120 | 17408 | iq4_kt |
| post_attention_norm | 5120 | 1 | f32 |
| attn_output | 6144 | 5120 | iq4_kt |
| nextn.shared_head_norm | 5120 | 1 | f32 |
| nextn.eh_proj | 10240 | 5120 | bf16 |
| nextn.enorm | 5120 | 1 | f32 |
| nextn.hnorm | 5120 | 1 | f32 |
| output | 5120 | 248320 | iq6_k |
| output_norm | 5120 | 1 | f32 |
Cheers
organization of Qwen - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your creation.
ApocalypseParty\zerofata - for the training effort.
bartowski - for the imatrix dataset + the myriad of quants we all benefit from.