Apex-35B-V1.0 GGUF
GGUF quants of Apexintelligence-AI/Apex-35B-V1.0 for some of the smaller sizes, made to lean slightly toward Japanese. For any other size, please use bartowski's quants, which measured well here. Thanks to bartowski for them and for the BF16 GGUF used as the reference below.
[!NOTE] Experimental. These files are evaluated only by KL divergence from BF16 on short Japanese and English texts; task benchmarks, long context and speed are not measured. The MTP layer is not run in the measurement.
Files
| File | Size | Mean KLD ja / en | vs bartowski IQ3_XXS (16.1 GB) |
|---|---|---|---|
| Apex-35B-V1.0-IQ3_XXS-exps-down3-hq.gguf | 15.6 GB | 0.2060 / 0.2832 | -6.5% / -7.5% |
Also here: imatrix.gguf (used for the file above), the quantization rule file
tensor-types-IQ3_XXS-exps-down3-hq.txt, and the llama-perplexity logs behind the table below
in kld/.
Comparison
KL divergence of the next-token distribution from BF16; lower is closer to the original. Sizes are in GB (10^9 bytes).
| File | Size | Mean KLD ja / en | 99.9% KLD ja / en | Same top token ja / en |
|---|---|---|---|---|
| Q8_0 (reference, see below) | 37.8 | 0.0197 / 0.0375 | 0.529 / 1.447 | 94.1% / 93.7% |
| bartowski Q4_K_L | 24.6 | 0.0473 / 0.0868 | 1.297 / 3.606 | 91.2% / 90.1% |
| bartowski Q4_K_M | 22.8 | 0.0572 / 0.0943 | 1.270 / 2.745 | 90.3% / 89.4% |
| bartowski Q4_K_S | 21.4 | 0.0637 / 0.0978 | 1.506 / 2.621 | 90.1% / 89.1% |
| bartowski IQ4_XS | 20.4 | 0.0728 / 0.0996 | 1.872 / 2.972 | 88.9% / 89.2% |
| bartowski IQ3_M | 19.4 | 0.1259 / 0.1922 | 2.530 / 4.328 | 85.1% / 84.9% |
| bartowski Q3_K_L | 18.7 | 0.1227 / 0.1906 | 2.672 / 4.884 | 85.5% / 85.1% |
| bartowski Q3_K_M | 17.8 | 0.1522 / 0.2088 | 2.499 / 4.276 | 83.7% / 83.9% |
| bartowski IQ3_XXS | 16.1 | 0.2203 / 0.3060 | 4.094 / 7.917 | 81.6% / 81.6% |
| IQ3_XXS-exps-down3-hq (this repo) | 15.6 | 0.2060 / 0.2832 | 4.154 / 5.296 | 82.2% / 81.7% |
How the file was made
- Layout (
tensor-types-IQ3_XXS-exps-down3-hq.txt, applied on top of the Q4_K_M defaults): every tensor outside the routed experts (attention, linear-attentionssm_*, shared expert,nextn.eh_proj) is Q8_0; the routed experts are IQ3_XXS for gate/up and IQ3_S for down. The MTP layer (blk.40) is Q4_0, as in bartowski's files. Token embedding is Q4_K and the output is Q6_K. - Imatrix (
imatrix.gguf): computed on a Q8_0 of this model over 367 chunks of 512 tokens from bartowski's calibration_datav3 and TFMC's ja/en C4 set (about 9% Japanese). Part of the data is wrapped in the ChatML chat format, including<think>and tool-call round trips, so that the chat and tool markers are part of the activations (llama-imatrix --parse-special -c 512). - llama.cpp build 11797 (commit
9b56ccf00d75), CPU backend:
llama-quantize --imatrix imatrix.gguf --tensor-type-file tensor-types-IQ3_XXS-exps-down3-hq.txt \
Apexintelligence-AI_Apex-35B-V1.0-bf16-00001-of-00002.gguf Apex-35B-V1.0-IQ3_XXS-exps-down3-hq.gguf Q4_K_M
Evaluation
- Reference: bartowski's
Apexintelligence-AI_Apex-35B-V1.0-bf16GGUF. llama-perplexity --kl-divergenceagainst BF16 logits, context 1024, CPU, on Japanese Wikipedia (wiki40b-ja test, 16 chunks) and English Wikipedia (wikitext-2 test, 8 chunks). Neither text is in the calibration data.- The Q8_0 row is
llama-quantize <BF16> Q8_0without an imatrix; it is not uploaded. - bartowski's files were downloaded on 2026-10-30.
kld/kld-base-*.logare the BF16 runs.
License and credits
Apache License 2.0, the same as the original model. Original model by Apexintelligence-AI; BF16 GGUF and the compared quants by bartowski; Japanese calibration data by TFMC.