niki-allocator: Qwen3.5-0.8B quantized models

These quants were produced by an experimental tool, niki-allocator. The tool is under active development and testing, and has not been published yet.

If you use these quants, I strongly recommend:

• Using the temperature and sampling settings recommended by Alibaba for Qwen3.5-0.8B
• Using this chat template: https://huggingface.co/Moore2877/Qwen-Fixed-Chat-Templates-llamacpp

Qwen3.5-0.8B quantization quality comparison

Per-tensor type allocation vs Unsloth Dynamic 2.0 quants, evaluated with per-chunk KL divergence against the BF16 model.

Quality vs size

Mean KLD as a function of model size shows the allocator curve consistently below the unsloth curve, with both corpora overlapping closely:

Models

• Model: Qwen3.5-0.8B (MTP/NextN, 25 layers: 24 main + 1 NextN head).
• Baseline: BF16 GGUF (1485.50 MiB), perplexity 6.055 on kilo-small corpus, 15.508 on wikitext-2 test.
• Per-tensor allocator quants (8 sizes): generated by a prototype per-tensor type allocator — each tensor assigned independently to the quantization type (Q5_K, Q6_K, IQ4_XS, etc.) that minimizes its proxy reconstruction error under a hard target size. Quantization performed by llama.cpp's own quantizer; imatrix-guided scale optimization used where the quantizer supports it. Source GGUF, imatrix, and assignment logs preserved.
• unsloth quants (8 sizes): unsloth/Qwen3.5-0.8B-MTP-GGUF, produced with Unsloth Dynamic 2.0 quantization (7 of 8 use the UD_ prefix: UD_Q2_K_XL, UD_Q3_K_XL, UD_Q4_K_XL, UD_Q5_K_XL, UD_Q6_K_XL, UD_IQ2_M, UD_IQ3_XXS; IQ4_NL has no prefix and is a more conventional quant), same model, MTP preserved.

Sizes were chosen to match: each allocator quant targets the size of an unsloth quant (e.g. allocator 511 MiB vs unsloth IQ4_NL 510.59 MiB).

Allocator type distributions vary by size: smaller targets use cheaper types (IQ4_XS, IQ3_S, Q3_K), larger targets transition to Q5_K/Q6_K/Q8_0. MTP/NextN tensors received types equal to or better than the unsloth quant's base type at the same size; quality gains are not achieved by sacrificing MTP heads.

Evaluation methodology

Metric: KL divergence per evaluation chunk between the quantized model's token probabilities and the BF16 baseline, computed by llama-perplexity with --saved-logits (full probability vectors saved, compared offline). Reported metrics: mean KLD, p95/p99/p99.9 KLD, maximum KLD (worst chunk), RMS Δp, same-top probability, top-flip weight, mean perplexity of the quant.

Corpora:

• kilo-small: ~290K tokens, hold-out from the calibration corpus (agent session transcripts: English, Russian, code). Used for the primary comparison (all 16 models evaluated).
• wikitext-2 test: ~290K tokens, standard test sample. Used for a subset of models to check generalization outside the calibration domain.

Chunk size: 2304 tokens per chunk (not the default 512). Reasoning models benefit from longer context windows during evaluation; 2304 was the fastest working size on the evaluation hardware. Reported KLD values are per-chunk averages and are not directly comparable to evaluations using 512-token chunks.

KLD bases: BF16 logits saved once per corpus; all quants compared against the same base.

Results: kilo-small corpus (all models)

* p99/p99.9 KLD are per-token percentiles over ~145K values (127 chunks × 1151 tokens per chunk). At 2304-token chunks, within-chunk KLD correlation is high: the effective tail sample for p99.9 is ~1-3 chunks, for p99 ~10-15 chunks. For ranking quants with a large gap (30%+), these metrics are reliable; for close quants (<10% in mean KLD), p99 and p99.9 carry high noise. Mean KLD is the primary metric; p95 is secondary.

At matched sizes, the allocator quant has lower mean KLD in every case. The gap is largest at small sizes (e.g. 390 MiB: 0.119 vs 0.378, −69%) and narrowest at the largest size (754 MiB: 0.0023 vs 0.0027, −15%). At 511 MiB vs unsloth IQ4_NL: mean KLD 0.0267 vs 0.0453 (−41%), max KLD 1.92 vs 2.80 (−32%).

Allocator 434 MiB (mean KLD 0.105) is worse than allocator 409 MiB (0.090), even though the allocator's proxy score was better at 434 MiB (total proxy loss 814 vs 1151 — the assignment was optimal for the proxy). The difference is in the token embedding tensor: at 409 MiB the allocator dropped it to IQ4_XS, freeing budget for more important tensors; at 434 MiB it kept it in Q5_K, spending extra budget where it mattered less. The same pattern appears in the unsloth lineup: unsloth UD_Q2_K_XL (433.52 MiB, KLD 0.326) is worse than the smaller unsloth UD_IQ3_XXS (409.23 MiB, KLD 0.241) — Q2_K is a known weak type, and unsloth also made a similar upgrade mistake further up the line (UD_Q6_K_XL → UD_Q5_K_XL on embeddings at a larger size step). Both dips are real (verifiable by GGUF paths in evaluation logs), not swapped data. They show that the proxy correlates with KLD but is not exact — the allocator's proxy is very cheap compared to real KLD runs, but it is an approximation, not a perfect substitute.

Results: wikitext-2 test (all models)

Same comparison on an unrelated corpus — all 16 models evaluated:

* Same caveat as for kilo-small: ~127 chunks, ~145K per-token KLD values, effective tail sample constrained by within-chunk correlation. Mean KLD is the primary metric.

Relative ranking preserved: allocator quants have lower mean KLD at matched sizes on an unrelated corpus. Absolute KLD values are higher (wikitext-2 is a harder domain for this model), but the allocator vs unsloth gap remains consistent. At 511 MiB vs unsloth IQ4_NL: mean KLD 0.0289 vs 0.0469 (−38%), max KLD 2.91 vs 4.04 (−28%).

Notes

• The allocator is a prototype, tested here on a single small model. Results on larger models may differ.
• Evaluation uses a single calibration corpus hold-out and one standard corpus; results reflect these specific distributions.
• Perplexity values are per-chunk means with 2304-token chunks; they are internally consistent but not directly comparable to PPL reported with different chunk sizes.
• Both the allocator and unsloth quants preserve MTP/NextN tensors; quality differences reflect type assignment strategy, not MTP handling.

Acknowledgments

• Alibaba and Qwen for the best open models that can run on consumer hardware
• Unsloth for quantized models used in comparisons and for setting a high quality bar
• michaelw9999 for ideas, inspiration, and advanced-gguf-quantizer