Nanbeige4.3-33B GGUF

GGUF quants of Nanbeige/Nanbeige4.3-33B, the 33B dense agentic and coding model from Nanbeige. Converted with llama.cpp from the official bf16 weights, one day after release. Tested on a single RTX 3060 12GB.

This is a dense model: all 33B params are used for every token. There is no MoE trick to keep experts in RAM, so for usable speed it has to fit in VRAM. With half of it on the CPU you get ~4 tok/s. That is what this quant is for.

Files

More sizes are on the way.

Run with llama.cpp

Single RTX 3060 12GB:

llama-server \
  -m Nanbeige4.3-33B-IQ2_XXS.gguf \
  -ngl 99 -c 24576 -fa on --jinja \
  -np 1 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  -b 2048 -ub 512 \
  --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
  --reasoning on

24576 is the max context that loads with everything on a 12 GB card (11.7 GB used). A 22K-token prompt runs fine at this setting. -ub 256 gets 26K at slower prefill. With q8_0 KV, 16K does not fit, so on 12 GB use q4_0 KV.

Sampling is the official one from the model card, temperature + top_p only (--top-k 0 --min-p 0.0 turn off llama.cpp's extra samplers):

Thinking is on by default. --chat-template-kwargs '{"enable_thinking":false}' turns it off. '{"preserve_thinking":true}' keeps old reasoning in multi-turn agent and coding work (the model card recommends it there). Tool calls use the Qwen3-Coder XML format, which llama-server --jinja parses into tool_calls.

Speed on RTX 3060 12GB

llama-bench, all layers on GPU, q4_0 KV, flash attention, -ub 512:

A real 22,300-token prompt: 410 tok/s prefill, 11.4 tok/s decode after it.

How this quant is built

The first version was a plain imatrix IQ2_XXS mix that copied the per-tensor pattern GSQ-RCO found for Qwen3.8-27B (more bits in the late layers). It answered fine, but on easy prompts it often skipped thinking and never emitted </think>, so the whole answer landed in reasoning_content and content was empty: 3 to 4 out of 8 runs on "write a python one-liner that reverses a string". Q4_K_M did it 0 out of 6. q4 vs q8 KV made no difference, and giving output.weight / token_embd more bits did not fix it either.

Swapping groups of layers in from Q4_K_M found where it lives (10 runs each):

So for Nanbeige the "think first, then close the tag" behavior sits in layers 6-12, not the late layers. This file keeps those 7 layers at Q4_K / Q6_K (from Q4_K_M) and the rest at 2-bit:

• layers 6-12: Q4_K / Q6_K
• attn_k / attn_v: Q4_K everywhere (tiny with 8 KV heads)
• late layers 39-51: attn_q / attn_output IQ3_XXS, ffn_down IQ2_S
• other attn_output: IQ2_XS, everything else IQ2_XXS
• output.weight: IQ4_XS, token_embd: Q2_K

Final check on this file with the config above: 12 / 12 runs reason and answer (temp 0.6 and 1.0), tool calls parse, strawberry has 3 r's.

The exact type of every tensor is in Nanbeige4.3-33B-IQ2_XXS.tensor-types.txt. imatrix: imatrix-Nanbeige4.3-33B.gguf (100 chunks x 512), computed on the Q4_K_M from bartowski's calibration_datav3 plus real agentic coding traces (multi-turn tool calls, rendered with this model's own chat template).

BOS fix

The original tokenizer config sets add_bos_token: true with <|im_start|> as the BOS token. Transformers never adds it in chat (the template already starts with <|im_start|>), but llama.cpp would, giving a doubled <|im_start|><|im_start|>system. This file sets tokenizer.ggml.add_bos_token = false, so prompts match the original model.

Credits

Model by Nanbeige. Quantized by @DogukanUrker.