Sage-A1.5-GGUF
GGUF quantizations of InternScience/Sage-A1.5, made with an importance matrix.
Files
| File | Size | Mean KLD ja / en | Mix | Use when |
|---|---|---|---|---|
Sage-A1.5-Q4_K_M.gguf |
21.7 GB | 0.0255 / 0.0227 | Standard Q4_K_M | Best quality here; you have the memory for it |
Sage-A1.5-IQ4_XS-exps.gguf |
19.3 GB | 0.0304 / 0.0264 | Experts IQ4_XS | Q4_K_M is a little too large; still better than the official Q4_K_M |
Sage-A1.5-IQ3_S-exps-down4.gguf |
17.5 GB | 0.0441 / 0.0370 | Experts gate/up IQ3_S, down Q4_K | Best 3-bit mix, about the official Q4_K_M's quality |
Sage-A1.5-Q3_K-exps-down4.gguf |
17.5 GB | 0.0463 / 0.0370 | Experts gate/up Q3_K, down Q4_K | Same size with K-quants only, which are usually faster when experts run on the CPU |
Sage-A1.5-IQ3_S-exps.gguf |
16.1 GB | 0.0576 / 0.0451 | Experts IQ3_S | 17.5 GB does not fit |
Sage-A1.5-Q3_K-exps.gguf |
16.1 GB | 0.0585 / 0.0477 | Experts Q3_K | Same, with K-quants only |
imatrix.gguf |
Importance matrix used for all files | |||
tensor-types-*.txt |
--tensor-type-file rules for the mixed files |
The mixed files (*-exps*) are Q4_K_M with the MoE expert tensors (ffn_{gate,up,down}_exps) changed; everything else keeps the Q4_K_M types. All files keep the built-in MTP (NextN) layer, so they work with --spec-type draft-mtp. For image input, use the vision projector from the official repo (Sage-A1.5-mmproj.gguf).
Quality
KL divergence (KLD) of the next-token distribution from BF16; lower is closer to the original. The official Q4_K_M, made without an imatrix, is included for comparison.
| Size | Mean KLD ja / en | 99.9% KLD ja / en | Same top token ja / en | PPL ratio ja / en | |
|---|---|---|---|---|---|
| Q8_0 (reference, not uploaded) | 37.8 GB | 0.0040 / 0.0047 | 0.127 / 0.196 | 97.3% / 97.4% | 1.000 / 0.997 |
| Q4_K_M | 21.7 GB | 0.0255 / 0.0227 | 0.538 / 0.494 | 92.9% / 93.8% | 1.015 / 1.014 |
| Q4_K_M, official | 21.7 GB | 0.0390 / 0.0378 | 1.070 / 0.978 | 90.9% / 92.0% | 1.019 / 1.036 |
| IQ4_XS-exps | 19.3 GB | 0.0304 / 0.0264 | 0.854 / 0.771 | 92.1% / 93.3% | 1.020 / 1.014 |
| IQ3_S-exps-down4 | 17.5 GB | 0.0441 / 0.0370 | 0.991 / 0.876 | 90.2% / 92.3% | 1.031 / 1.025 |
| Q3_K-exps-down4 | 17.5 GB | 0.0463 / 0.0370 | 1.716 / 1.148 | 90.4% / 92.2% | 1.035 / 1.017 |
| IQ3_S-exps | 16.1 GB | 0.0576 / 0.0451 | 1.482 / 1.065 | 89.5% / 91.3% | 1.040 / 1.037 |
| Q3_K-exps | 16.1 GB | 0.0585 / 0.0477 | 1.805 / 1.795 | 89.3% / 91.2% | 1.047 / 1.036 |
- At the same size, Q4_K_M with the imatrix has 35-40% lower KLD than the official Q4_K_M.
- IQ4_XS-exps is 2.4 GB smaller than Q4_K_M and still 22-30% below the official Q4_K_M in mean KLD.
- Keeping
ffn_down_expsat Q4_K costs 1.4 GB and removes most of the gap between the 3-bit mixes and the official Q4_K_M: the-down4files are within about 0.98-1.19x of its mean KLD. - IQ3_S and Q3_K have about the same mean KLD at the same size, but IQ3_S has a much lower tail (99.9% KLD). IQ3_S-exps-down4 is the only file below 18 GB whose tail is below the official Q4_K_M's.
How these were made
- llama.cpp
9b56ccf(CPU build). BF16 GGUF converted from the original safetensors; vocabulary and chat template are identical to the official GGUF. - Imatrix: computed on Q8_0 over 367 chunks of 512 tokens from bartowski's calibration_datav3 and TFMC's ja/en C4 set. About a third of the chunks are wrapped in Sage's chat format, including
<think>and tool-call round trips, so that the chat and tool markers are part of the activations. The MTP layer is not run during imatrix collection and is quantized without it, as in the official file. - Mixed files: Q4_K_M plus the rules in
tensor-types-<name>.txt. The expert tensors hold about 94% of the weights; attention, the linear-attention (SSM) tensors, the shared expert and the output keep the Q4_K_M types. The rules cover layers 0-39 only, so the MTP layer (blk.40) keeps the Q4_K_M mix. Q3_K-exps follows the MiniCPM-V 4.7 35B-A3B mix.
llama-quantize --imatrix imatrix.gguf --tensor-type-file tensor-types-IQ3_S-exps-down4.txt \
Sage-A1.5-BF16.gguf Sage-A1.5-IQ3_S-exps-down4.gguf Q4_K_M
- KLD:
llama-perplexity --kl-divergenceagainst BF16 logits, context 1024, on Japanese Wikipedia (wiki40b-ja test, 16 chunks) and English Wikipedia (wikitext-2 test, 8 chunks). Neither text is in the calibration data.
License and credits
Apache License 2.0, the same as the original model.