Quality-first GGUF quants of OrionLLM/GRM-2.6-Plus-0628

Source model: https://huggingface.co/OrionLLM/GRM-2.6-Plus-0628

These quants were produced by my experimental tool, niki-allocator. The main difference of my method is per-tensor type allocation (not per tensor groups, layers, community lore etc.) Each tensor gets what is predicted to be best for it.

The tool is under active development and testing, and has not been published yet.

Quantization quality comparison

To prove my method I've performed comparison with quants from other providers:

• bartowski
• DAXZEIT

Quants for comparison were downloaded on August 4, 2026.

Lower is better

Sizes

My quant sizes were chosen to fill 8500-23000 MiB with geometric progression and rounded to 100 MiB.

No special size choosing was performed (there may be exact sizes that are better, but they are left on the table due to hardware constraints).

niki-allocator is able to produce allocation with ~1 MiB precision.

Which one to choose?

The bigger - the better. So it is always a compromise of what you can afford considering context size and other constraints. Choose the biggest you can fit.

Quant names

Hugging Face requires quant type in the filename to properly show it. But all my quants are mixed by design. I decided to include dominating quant type (by size, not by tensor count) to the name therefore. It may easily be less than 50% of the model though. So choose by size, ignore listed quant type. For my quants: bigger --> better. Always.

Methodology

niki-allocator

It is quality-first per-tensor quantization allocator for llama.cpp quantize.

Allocator decides, per tensor, which quant type to use so that the resulting GGUF hits an exact target file size with the best estimated quality. It is not direct measurement of KLD-metrics, it is only estimation. But as you can see from real KLD evaluations estimation is good enough. Raw results are already good. But I added some calibration levers on top to make it better (for Qwen3.5-0.8B I used uncalibrated estimation). These levers need series of KLD evaluations to move estimation closer to reality.

niki-allocator is work in progress. Not published yet.

Corpora

• imatrix: niki-small corpus - ~330K tokens from my own agentic sessions (English, Russian, code, reasoning). Deduped, cleaned, filtered by simhash similarity to include only really different turns, short turns (less than 16K chars) dropped. Used for imatrix only.
• Calibration of niki-allocator and evaluation: niki-small-plus corpus - ~405K tokens from my own agentic sessions (English, Russian, code, reasoning). Deduped, cleaned, filtered by simhash similarity to include only really different turns, short turns (less than 12K chars) dropped. Does not include any exact or similar (by simhash) agentic turns from niki-small.
• Independent evaluation: wikitext-2 test corpus - ~290 K tokens standard corpus. It wasn't seen during calibration or imatrix at all. Only for final evaluation before publishing. So my quants are not anyhow tuned for this dataset.

Chunk sizes

• imatrix (niki-small): 8064 tokens (384 x 21). To catch reasoning and long turn traces, not just basic chat.
• niki-small-plus evaluation: 6144 tokens (384 x 16). 66 chunks total. To catch reasoning and long turns, but to keep chunks amount somewhat reasonable (that's why lower than for imatrix).
• wikitext-2 test evaluation: 768 tokens (384 x 2). 386 chunks total. It is not directly comparable with popular 512 tokens chunks! Why 384 magic number? My hardware is fastest with ubatch 384, so I use it.

Metrics

All metrics were evaluated with llama-perplexity against Q8_0 standard quant (without imatrix) due to hardware constraints.

Quantization was performed with patched llama-quantize from BF_16 source. Fully compatible with original llama.cpp.

All my and foreign quants were evaluated against the same base.

llama.cpp commit 876a4321163249c43ca4e986818fab5ab081f282.

The main metric during calibration of niki-allocator was mean KLD. Also I used p95 KLD, PPL and same top p as secondary metrics.

Better PPL doesn't always mean better other more important metrics. So it can't be used as a main metric (some people still use). It is just fast indicator (no need for logits base) to check for big problems.

Why not p99 KLD, p99.9 KLD, max KLD? With such a small corpus (well, almost everybody seems to use wikitext-2 test with only 290K tokens) these metrics are not very robust. While p99 KLD can give some plausible comparison for the allocation method as a whole (that's why I include graphs of it), p99.9 is essentially a corpus-driven noise (too little data), max KLD is just random at any corpus size (one hardest token of the whole dataset, saying nothing about other hard tokens). Please correct me if I am wrong.

Evaluation results

niki-small-plus corpus

wikitext-2 test corpus

MTP

I've tested MTP performance and discovered that there is no point to keep MTP head higher than IQ4_XS. Q3_K is near but slightly worse (used for lower quants). Exact allocation of IQ4_XS/Q3_K is chosen by allocator (not pinned). Lower than that there is considerable quality drop, so not used in my quants.

Speculation parameters: --spec-draft-n-max 5 --spec-draft-p-min 0.8.

Standard llama.cpp speed-bench with all 11 categories x 5 samples, osl 128, concurrency 1, temperature 0.

GPUs: 2 x 5060Ti

Chat template

Chat template included with gguf's is v16 from here: https://huggingface.co/Moore2877/Qwen-Fixed-Chat-Templates-llamacpp

I believe it is good.

Multimodality

It works fine with original Qwen mmproj. I use this one: https://huggingface.co/DAXZEIT/Qwen3.6-27B-mmproj-hybrid-Q8_0-F16-gguf

Acknowledgments

• OrionLLM - great source model
• Qwen - great base model which we all are tinkering with
• DAXZEIT - quants for comparison
• bartowski - quants for comparison
• llama.cpp - well, you know them already
• Moore2877 - chat template
• michaelw9999 - some ideas to start with