G9v3-39A5B — Q2 GGUF
Fast GGUF for mainstream llama.cpp. Requires a build with G9v3 architecture
support (e.g. linuxid10t/llama.cpp@feature/g9v3-support) until upstream
llama.cpp merges it — standard builds will not load this model.
Recipe (antirez style)
| Tensor group | Format |
|---|---|
| Routed expert gate / up | IQ2_XXS (imatrix) |
| Routed expert down | Q2_K |
| Shared experts, attention, output | Q8_0 |
| Token embeddings | F16 |
| Routers, norms, biases | F32 |
Same strategy as Salvatore Sanfilippo's
DeepSeek V4 quants:
imatrix-calibrated IQ2_XXS gate/up with K-quant down routed experts, Q8_0
protection everywhere else. Imatrix:
linuxid10t.
Files
| File | Size |
|---|---|
| G9v3-39A5B-Q2-IQ2_XXS-Q2_K.gguf | 11.21 GiB |
Validation
gguf-dump audit: all routed gate/up are IQ2_XXS, all down are Q2_K
(native 256-wide rows, no padding needed), zero misquantized experts;
attn/shexp/output Q8_0, token F16, norms/router F32. Spot ppl + KL
divergence vs the Q8_0 reference on short prompts (arithmetic, Italian
fact, C completion, counting, colors), CPU-only: Q8 PPL 5.21, Q2 PPL 6.77,
KL 0.21, same top token 83.9%.
Usage
# needs a G9v3-enabled llama.cpp build, e.g.
# git clone -b feature/g9v3-support https://github.com/linuxid10t/llama.cpp
llama-cli -m G9v3-39A5B-Q2-IQ2_XXS-Q2_K.gguf -t 16 -c 32768
# server
llama-server -m G9v3-39A5B-Q2-IQ2_XXS-Q2_K.gguf -t 16 -c 32768 --host 0.0.0.0 --port 8080
Model
G9v3-39A5B is a 320-expert MoE preview model by AI9Stars. Context: long.
Original model: ai9stars/G9v3-39A5B. Original BF16 GGUF: linuxid10t/G9v3-39A5B-GGUF. License: Apache-2.0.