Nanbeige4.3-33B — AutoRound INT4 (W4A16)
An INT4 AutoRound quantization of
Nanbeige/Nanbeige4.3-33B, a dense 33B reasoning
and agentic model (LlamaForCausalLM, 52 layers, GQA 56 / 8 heads, 512K context). It is 20.8 GB
and runs on two 24 GB cards with ~200K tokens of fp8 KV cache.
It was made to serve the model on two RTX 3090s in club-3090, with the same AutoRound recipe as our other INT4 builds.
Quantization
| Scheme | W4A16: int4, symmetric, group size 128, auto_round:auto_gptq packing |
| Quantized | every linear layer in the 52 decoder layers (364) |
| Kept in BF16 | lm_head, embeddings, all norms |
| Tool | auto-round 0.14.2 |
| Tuning | 200 iterations, 512 samples, seqlen 2048, enable_alg_ext=True |
| Calibration | auto-round's default (NeelNanda/pile-10k) |
| Source | Nanbeige/Nanbeige4.3-33B @ c1093c50 |
The recipe is also stored in club3090_build_recipe.json. The tokenizer files, chat template and
generation_config.json are the original model's, byte for byte.
The original checkpoint has no MTP head, so neither does this one.
Serving
⚠️ Needs transformers 5.x. config.json keeps the RoPE theta (1.2e8) only in rope_parameters, as
transformers 5 writes it. A stack on transformers 4.x finds no rope_theta and silently falls back to
10,000, which garbles the output. vLLM v0.32.0 (transformers 5.19) and SGLang v0.5.22 (transformers
5.19) both read it correctly.
vLLM (the AutoRound config resolves to vLLM's inc method on its own):
vllm serve wasifb/Nanbeige4.3-33B-AutoRound-W4A16 \
--tensor-parallel-size 2 --max-model-len 204800 --kv-cache-dtype fp8_e4m3 \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
SGLang:
python -m sglang.launch_server --model-path wasifb/Nanbeige4.3-33B-AutoRound-W4A16 \
--tp-size 2 --quantization auto-round --context-length 204800 --kv-cache-dtype fp8_e4m3 \
--chunked-prefill-size 2048 --reasoning-parser qwen3 --tool-call-parser qwen3_coder
On 24 GB cards keep SGLang's --chunked-prefill-size at 2048: 8192 ran out of memory while capturing
the prefill CUDA graphs.
The parsers are the ones the original card uses. Its tool format defaults to XML
(tool_call_format="xml", parsed by qwen3_coder); json works with hermes. Thinking is on by
default (enable_thinking), and preserve_thinking is left to the template, which keeps earlier
reasoning: the original card recommends that for agents and false for chat. Sampling follows the
original card: agentic and tool use temperature=1.0, top_p=0.95, reasoning and chat 0.6 / 0.95.
Validation
Tested on 2× RTX 3090 (PCIe), tensor parallel 2, fp8 KV, 204,800 context:
| vLLM v0.32.0 | SGLang v0.5.22 | |
|---|---|---|
| Loads (Marlin int4 kernels) | yes, 9.83 GiB a card | yes, 10.11 GB a card |
| fp8 KV pool | 207,616 tokens | 206,433 tokens |
| verify-full (answers, tool calls, streaming tool calls, reasoning split, a long answer without repetition) | pass | pass |
These are serving checks. Throughput figures and the quality evaluation (club-3090's 8-pack) are pending, and this card will be updated with them.
License
Nanbeige4.3-33B is licensed under Apache-2.0 (see LICENSE). This quantization is a derivative and
is distributed under the same license.
All credit for the model goes to the Nanbeige team. This repository only adds the quantization.