Experiment 29: Qwen/Qwen3-8B quality benchmark
- Status:
completed - Model:
Qwen/Qwen3-8B - Revision:
b968826d9c46dd6066d109eabc6255188de91218 - Candidate run:
/workspace/NanoQuant/evidence/029/029-compress-and-benchmark-qwen3-8b - Backend:
llama.cpp - Wall time: 1769.80 seconds
completed means all evaluators returned finite metrics; it is not a BF16-quality acceptance gate.
Protocol
- WikiText-2: 64 samples × 128 tokens, batch 8
- WikiText token hash:
sha256:e85e3ae8da01294229ac7dfc651f5d44f74eadaff974ad80a5a19779b58fbe0f - Tasks: piqa, arc_easy, arc_challenge, hellaswag, winogrande, boolq; first 200 rows, batch 4
- Tokenizer hash:
sha256:af4e928cfa630ef78563f4800a751d568307a7cd4d4f5fd5919de27b0fbfdbc4
Quality results
| Benchmark | Metric | BF16 | NanoQuant | Delta | Ratio |
|---|---|---|---|---|---|
| WikiText-2 | perplexity ↓ | 23.110475 | 40.599726 | +17.489251 (+75.68%) | 1.7568x |
| piqa | acc_norm ↑ | 0.8000 | 0.5350 | -0.2650 | 0.6687x |
| arc_easy | acc_norm ↑ | 0.8050 | 0.4000 | -0.4050 | 0.4969x |
| arc_challenge | acc_norm ↑ | 0.5350 | 0.3400 | -0.1950 | 0.6355x |
| hellaswag | acc_norm ↑ | 0.6350 | 0.4050 | -0.2300 | 0.6378x |
| winogrande | acc ↑ | 0.7050 | 0.5700 | -0.1350 | 0.8085x |
| boolq | acc ↑ | 0.8550 | 0.6600 | -0.1950 | 0.7719x |
Deployment storage
| Deployable representation | Bytes | Storage savings vs BF16 |
|---|---|---|
| BF16 checkpoint tensors | 16,381,516,776 | — |
| NanoQuant GGUF | 2,214,297,824 | 86.48% |
The packed NanoQuant quantized-layer payload is 884,765,160 bytes. The complete GGUF is the comparable deployable artifact and also includes embeddings, the output projection, norms, and metadata.
Quality execution
The BF16 reference uses Transformers. The NanoQuant candidate is evaluated only from the exported GGUF by the pinned llama.cpp runtime; no reconstructed dense or factorized PyTorch candidate is loaded.
| Runtime | Elapsed seconds | Peak dedicated GPU bytes | Peak host bytes |
|---|---|---|---|
| BF16 Transformers | 34.98 | 18,886,950,912 | 4,983,508,992 |
| NanoQuant llama.cpp GGUF | 1731.46 | unavailable | 2,550,489,088 |
- GGUF:
/workspace/NanoQuant/Results/029/qwen3-8b-nanoquant.gguf - GGUF SHA-256:
sha256:75d3f4f150195e0c6a7ccd2ce3372f919534fb7ec609f6b9ba52eb71af194253 - llama.cpp commit:
2927569f318311aded148d76be6b1109664066f5
Provenance
- Experiment config hash:
sha256:993eca2ef6c345d01926e4677e9cf9620c001fa7e61f1ffa207eeaad1e1803cb - Launcher:
experiments/029-compress-and-benchmark-qwen3-8b.py - Candidate identity:
{"config_hash":"sha256:9356ea7a7e46c6dc5286ce574c110add86e4fd77b9f0c92dc32f4646888f9dd9","model_hash":"sha256:db9a057024370dc56a8bba76b9f274692c2094615dc4da1497ce830d14707bc6","plan_hash":"sha256-bc87a0d973d42d9bc247fd72660dc76c590ffe29fbf918f1aa8db56e87c4dd8a"} - Global tuning:
None