imatrix GGUF SuiteImportance-matrix (llama-imatrix) GGUF quantizations of Qwen/Qwen2.5-Coder-7B-Instruct, calibrated specifically for agentic tool-calling, code review, and declarative infrastructure reasoning and benchmarked across 8 GiB NVIDIA Jetson Orin Nano (CUDA 12.6 sm_87) and x86_64 AVX2.
spaces/kleinbem/edge-llm-paretodatasets/kleinbem/edge-llm-benchmarksgithub.com/kleinbem/hf-benchUnlike perplexity-only tables, every quantization below was scored directly against the exact FP16 base model probability distribution (llama-perplexity --kl-divergence), alongside real hardware throughput and INA3221/RAPL power telemetry:
| Quant | imatrix |
BPW | Size (GiB) | Mean KLD (nats) | Top-1 Match | PPL | Orin CUDA tg128 |
Orin Power / Energy | x86 AVX2 tg128 |
|---|---|---|---|---|---|---|---|---|---|
Q8_0 |
Yes | 8.50 | 7.54 | 0.00026 | 99.61% | 8.276 | OOM (>8 GiB) | — | 2.2 tok/s |
Q6_K |
Yes | 6.57 | 5.83 | 0.00121 | 98.69% | 8.278 | OOM (>8 GiB) | — | 2.6 tok/s |
Q5_K_M |
Yes | 5.72 | 5.07 | 0.00286 | 97.97% | 8.268 | OOM (>8 GiB) | — | 2.0 tok/s |
Q4_K_M |
Yes | 4.92 | 4.36 | 0.01005 | 96.08% | 8.356 | 13.0 tok/s | 20.4 W (1.570 J/tok) | 2.8 tok/s |
Q4_K_M-std |
No (control) | 4.92 | 4.36 | 0.01321 | 95.49% | 8.281 | 13.0 tok/s | 20.3 W (1.565 J/tok) | 3.0 tok/s |
IQ4_XS |
Yes | 4.43 | 3.93 | 0.01287 | 95.36% | 8.307 | 16.3 tok/s | 20.5 W (1.256 J/tok) | 2.4 tok/s |
IQ3_M |
Yes | 3.75 | 3.33 | 0.04187 | 90.52% | 8.374 | 11.9 tok/s | 20.9 W (1.756 J/tok) | 1.1 tok/s |
imatrixAblation (Q4_K_MvsQ4_K_M-std): Calibrating second-moment channel activations on agentic tool-calling + systems code changed Mean KL-Divergence from0.01321to0.01005(+23.9% KLD reduction at identical byte size).
Qwen2.5-Coder-7B-Instruct uses Grouped-Query Attention with 28 layers, 4 KV heads, and head_dim = 128 (56.0 KiB/token in FP16 KV cache, or 448 MiB at 8,192 context). Use the interactive Edge Unified-Memory Sizer to verify exact headroom for your context length and KV quantization (FP16 / Q8_0 / Q4_0).
nix run github:kleinbem/hf-bench#hf-bench -- run-local --model qwen2.5-coder-7b
nix run github:kleinbem/hf-bench#hf-bench -- bench-orin --model qwen2.5-coder-7b --host orin-nano
Qwen/Qwen2.5-Coder-7B-Instruct