What's that?
The goal: Make a mixed precision quant of TheDrummer/Big-Alice-28B-v1 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit in 16GB VRAM with at least 16k tk of KV cache and system overhead.
The result: Mixed precision quantization of TheDrummer/Big-Alice-28B-v1 utilizing higher precision 3-bit quants for ffn and 4-bit trellis for most attention tensors.
quantized with ik_llama.cpp build: 9d07d868
incompatible with mainline llama.cpp
Layout
| Layer | Dims | Dims | Quant |
|---|---|---|---|
| token_embd | 5120 | 131072.0 | iq4_kt |
| blk | 40 | ||
| attn_k | 5120 | 1024 | iq4_kt |
| attn_norm | 5120 | 1 | f32 |
| attn_q | 5120 | 4096 | iq4_kt |
| attn_v | 5120 | 1024 | iq4_kt |
| attn_output | 4096 | 5120 | iq4_k |
| ffn_down | 14336 | 5120 | iq3_k |
| ffn_gate | 5120 | 14336 | iq3_k |
| ffn_norm | 5120 | 1 | f32 |
| ffn_up | 5120 | 14336 | iq3_k |
| output | 5120 | 131072 | iq5_k |
| output_norm | 5120 | 1 | f32 |
Conclusion
It works! Kinda. KV cache for upscaled Apriel is still tremendous, and I can only fit 20960tk in q4_0/q4_0 with system overhead. Candidly, I didn't notice too much difference from Snowpiercer, which outputs a very similar quality text without eating a neighbourhood worth of VRAM to produce it.
Regardless, since the quant was made, it is provided. WYSIWYG.
Cheers
SillyTilly - for the base model.
TheDrummer - for the finetuned model.
mradermacher - for the imatrix + the myriad of quants we all benefit from.
ikawrakow and contributors of ik_llama.cpp - I probably misused your ingenious creation.