Model Card for Model ID

This is a IQ4_KT quantization of Naphula/Slimaki-Tavern-24B-v1.3, created specifically to challenge mainline IQ4_XS quants by being smaller and not being dumber.

Quant Details

iq4_kt: attn_q, attn_v, attn_k, ffn_down, ffn_gate, ffn_up, attn_output (280)
iq4_k: token_embd (1)
iq6_k: output (1)
imatrix by mradermacher

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp
only tested with ik_llama.cpp (though I expect croco.cpp to work as well?)

Comparison:

IQ4_XS by mradermacher:

winogrande: 74.3489 ± 1.2274 (942/1267)
hellaswag (n=400, s=42): 82.25%

with K q5_0, V q4_1:
pp512: 1457.63 ± 157.65
pp32768: 1108.26 ± 5.68
tg128: 31.27 ± 0.79
tg512: 31.54 ± 0.16
size: 12758946304 bytes

this IQ4_KT:

winogrande: 74.7435 ± 1.2211 (947/1267) (+0.53%, statistically insignificant)
hellaswag (n=400, s=42): 83.25% (+1.2%, statistically insignificant)

with K q5_0, V q4_1:
pp512: 1168.07 ± 30.28 (19.8% slower)
pp32768: 955.13 ± 4.38 (13.8% slower)
tg128: 32.39 ± 0.22 (3.58% faster)
tg512: 31.52 ± 0.26 (0.07% slower)
size: 12070818560 bytes (5.39% smaller) (-656 MB)

In conclusion

-Is it smarter?
Not really.

-Is it slower?
Yeah.

-Is it smaller?
Yeah. 656 MB is about 12k extra context at K(Q5_0) V(Q4_1).

-Is it worth it?
No idea. I made it for personal use to squeeze some extra context in without losing brainpower. I can take the PP speed hit. WYSIWYG.

-Possible 12G VRAM full fit with KVO?
Probably. A mixed IQ3_KT ffn / IQ4_KT attn would likely fit with some context headroom. It's going to be slower. It's probably not going to be THAT much dumber.

-Why Q5_0/Q4_1?
Reasons

Cheers

MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
Naphula - you cooked a sick one.
Everyone whose finetunes were included in the merge!
mradermacher - for the imatrix + the myriad of quants we all benefit from.