imatrix GGUF SuiteImportance-matrix (llama-imatrix) GGUF quantizations of Qwen/Qwen3-4B, calibrated specifically for agentic tool-calling, code review, and declarative infrastructure reasoning and benchmarked across 8 GiB NVIDIA Jetson Orin Nano (CUDA 12.6 sm_87) and x86_64 AVX2.
spaces/kleinbem/edge-llm-paretodatasets/kleinbem/edge-llm-benchmarksgithub.com/kleinbem/hf-benchUnlike perplexity-only tables, every quantization below was scored directly against the exact FP16 base model probability distribution (llama-perplexity --kl-divergence), alongside real hardware throughput and INA3221/RAPL power telemetry:
| Quant | imatrix |
BPW | Size (GiB) | Mean KLD (nats) | Top-1 Match | PPL | Orin CUDA tg128 |
Orin Power / Energy | x86 AVX2 tg128 |
|---|---|---|---|---|---|---|---|---|---|
Q8_0 |
Yes | 8.52 | 3.99 | 0.00111 | 98.89% | 14.501 | 15.2 tok/s | 18.6 W (1.224 J/tok) | 3.4 tok/s |
Q6_K |
Yes | 6.58 | 3.08 | 0.00494 | 97.65% | 14.326 | 14.4 tok/s | 19.7 W (1.365 J/tok) | 3.9 tok/s |
Q5_K_M |
Yes | 5.75 | 2.69 | 0.01373 | 96.01% | 14.616 | 16.9 tok/s | 19.9 W (1.174 J/tok) | 3.7 tok/s |
Q4_K_M |
Yes | 4.97 | 2.33 | 0.04804 | 92.16% | 14.571 | 18.8 tok/s | 20.1 W (1.069 J/tok) | 4.7 tok/s |
Q4_K_M-std |
No (control) | 4.97 | 2.33 | 0.08262 | 91.89% | 15.746 | 18.8 tok/s | 19.6 W (1.042 J/tok) | 4.6 tok/s |
IQ4_XS |
Yes | 4.52 | 2.12 | 0.07572 | 92.81% | 15.318 | 22.4 tok/s | 19.1 W (0.853 J/tok) | 4.0 tok/s |
IQ3_M |
Yes | 3.91 | 1.83 | 0.15560 | 87.52% | 14.579 | 17.8 tok/s | 20.4 W (1.145 J/tok) | 2.4 tok/s |
imatrixAblation (Q4_K_MvsQ4_K_M-std): Calibrating second-moment channel activations on agentic tool-calling + systems code changed Mean KL-Divergence from0.08262to0.04804(+41.9% KLD reduction at identical byte size).
Qwen3-4B uses Grouped-Query Attention with 36 layers, 8 KV heads, and head_dim = 128 (144.0 KiB/token in FP16 KV cache, or 1152 MiB at 8,192 context). Use the interactive Edge Unified-Memory Sizer to verify exact headroom for your context length and KV quantization (FP16 / Q8_0 / Q4_0).
nix run github:kleinbem/hf-bench#hf-bench -- run-local --model qwen3-4b
nix run github:kleinbem/hf-bench#hf-bench -- bench-orin --model qwen3-4b --host orin-nano
Qwen/Qwen3-4B