AH-Nanbeige4.3-33B IQ3_XXS (GGUF)
An IQ3_XXS quantization of Nanbeige4.3-33B by AH, made with bartowski's imatrix. A 33B agent model in one 13.1 GB file. Our goal is high accuracy for agents on small, low-cost systems: tool calls, terminal work and code on one consumer GPU.
This file is a quantization of Nanbeige/Nanbeige4.3-33B by the Nanbeige team. We did not change the weights by training. Their work makes this release possible. The importance matrix (imatrix) is from bartowski. Thank you, bartowski.
How did we make it?
We quantized it from bartowski's Q8_0 with bartowski's imatrix, and we used a Q5_K output tensor so that Kindi can run it. Then we measured each candidate file against the original model, token by token. We kept the file that stayed closest to the original at this size.
Against the original model on our held-out text:
| Measure | Value |
|---|---|
| Same top token as the original | 82.8% |
| KL divergence from the original | 0.225 |
| File size | 13.1 GB |
How does it score?
This is our initial test: 115 tasks, one seed per task, the same settings for both models.
- kbench: 65 terminal and coding tasks for agents. The model works in the pi agent harness, and hidden tests check each result.
- 50 recent LiveCodeBench problems.
| Test | Nanbeige 33B IQ3_XXS (this file, llama.cpp) | Bonsai 27B (Kindi) |
|---|---|---|
| kbench, 65 agent tasks | 64 (98.5%) | 64 (98.5%) |
| LiveCodeBench, 50 problems | 34 (68.0%) | 32 (64.0%) |
| All 115 tasks | 98 (85.2%) | 96 (83.5%) |
Both models are almost at the limit of kbench. The model has much more room to show its strength on harder tests. We are making those tests now.
How do I run it?
With llama.cpp (use --jinja for the chat template and tool calls):
llama-server -m AH-Nanbeige4.3-33B-IQ3_XXS.gguf --jinja -ngl 99 -c 16384 \
-ctk q8_0 -ctv q8_0 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0
On one RTX 3090, llama.cpp writes about 35 tokens per second with this file.
What about Kindi?
Kindi is our custom engine that runs modern LLMs fast on low-cost NVIDIA cards, from a 1 GB Quadro K420 to an RTX 3080. Kindi support for this model is in development. Our target is the same output as llama.cpp, at a higher speed. Follow amirhusain.com/kindi for the release.
License and credits
- Base model: Nanbeige/Nanbeige4.3-33B, Apache License 2.0. This file has the same license. The change is the quantization to IQ3_XXS.
- Quantization format: llama.cpp.
- Released by Amir Husain.