Nanbeige4.3-33B GSQ-RCO GGUF
Mixed-precision GSQ-RCO GGUF quantizations of Nanbeige/Nanbeige4.3-33B, optimized for efficient, high-fidelity local inference in llama.cpp.
Benchmark
GGUF Files
| File | bpw | Size |
|---|---|---|
| Nanbeige4.3-33B-GSQ-RCO-IQ2_XXS.gguf | 2.40 | 9.9 GB |
| Nanbeige4.3-33B-GSQ-RCO-IQ2_XS.gguf | 2.65 | 10.9 GB |
| Nanbeige4.3-33B-GSQ-RCO-IQ2_M.gguf (Recommended) | 3.05 | 12.7 GB |
| Nanbeige4.3-33B-GSQ-RCO-IQ3_XS.gguf | 3.60 | 14.8 GB |
| Nanbeige4.3-33B-GSQ-RCO-IQ3_M.gguf | 3.90 | 16.1 GB |
| Nanbeige4.3-33B-GSQ-RCO-IQ4_XS.gguf | 4.50 | 18.7 GB |
| Nanbeige4.3-33B-GSQ-RCO-Q5_K_M.gguf | 5.75 | 23.8 GB |
Usage
llama.cpp
Interactive Chat
llama-cli \
-m Nanbeige4.3-33B-GSQ-RCO-IQ2_M.gguf \
-ngl 99 \
-fa \
-b 2048 \
-c 16384 \
-cnv
Local Server (OpenAI-Compatible)
llama-server \
-m Nanbeige4.3-33B-GSQ-RCO-IQ2_M.gguf \
-ngl 99 \
-fa \
-b 2048 \
-c 32768 \
--port 8080
64k Long-Context (Low-VRAM)
Using 4-bit KV cache (-ctk q4_0 -ctv q4_0) allows 65k context to run within < 12 GB total VRAM on consumer 12 GB GPUs (e.g. RTX 3060 12GB, RTX 4070):
llama-server \
-m Nanbeige4.3-33B-GSQ-RCO-IQ2_XXS.gguf \
-ngl 99 \
-fa \
-b 2048 \
-c 65536 \
-ctk q4_0 -ctv q4_0 \
--port 8080
Ollama
ollama run hf.co/abenzerps/Nanbeige4.3-33B-GSQ-RCO-GGUF:IQ2_M
Attribution & License
- Distributed under the Apache-2.0 License, matching upstream Nanbeige/Nanbeige4.3-33B.
- Base model: Nanbeige/Nanbeige4.3-33B.