Nanbeige4.3 33B, EXL3

Nanbeige4.3-33B is Nanbeige's dense model for agents and coding. This is a 4.0 bits-per-weight EXL3 quantisation of Nanbeige/Nanbeige4.3-33B, made by Yamz Labs. It runs on one 24 GB NVIDIA card with the Kyojin engine, up to about 77K tokens of context.

Model size: 33.1B parameters. The pack is 18.7 GB (17.4 GiB) in 3 shards. The parameter count in the Hugging Face sidebar is not a model figure: the site counts the packed 16-bit words of the EXL3 format.

Hardware: tested on one NVIDIA RTX 3090 (24 GB) with the Kyojin engine. The pack is in the standard EXL3 format; other cards and other runtimes are untested.

Highlights

• Fits a 24 GB card. With a 6-bit KV cache the server holds 77,824 tokens for one user, or 3 users at 24K each.
• Quality. Against the original BF16 model, KL is 0.086 with top-1 agreement of 90.6 % over six kinds of text (see Quality).
• Speed on one RTX 3090. Decode 36.7 tok/s for one user at 8K of context, 108 tok/s in total for 4 users. Prefill 1380 tok/s.
• Agent work. 24 of 30 multi-turn agent tasks fully solved and 10 of 14 harder ones, through a real agent client with shell and file tools (see Evals).
• Tool calls and thinking. The Kyojin server parses the model's <think> blocks into reasoning_content and its tool calls into OpenAI tool_calls.
• Feedback wanted. Run it on your own card and tell us your tok/s: open an issue at https://github.com/Yamz-Labs/kyojin/issues or start a discussion on this page.

How it was made

This pack is our own conversion. It is not a copy of anyone else's quantisation. We ran the EXL3 conversion of Kyojin (the Yamz engine built on ExLlamaV3) on the BF16 checkpoint of Nanbeige/Nanbeige4.3-33B:

• Bit widths. All linear layers at 4.0 bits per weight, output head at 6 bit, embeddings in fp16.
• Calibration. The standard EXL3 set: 250 rows of 2048 tokens.

This is a plain conversion with no per-layer bit allocation. The conversion tooling is not published. The engine and these weights are.

Run

Kyojin 1.7.3 or later. Build the NVIDIA extension (CUDA 13.0, torch 2.10 cu130, gcc 13, details in tools/cuda/README.md):

git clone https://github.com/Yamz-Labs/kyojin && cd kyojin
tools/cuda/build.sh /path/to/cuda13 /path/to/venv-cuda/bin/python
python tools/qwen/serve.py --model /path/to/this-pack --port 8000 \
  --chat-template /path/to/this-pack/chat_template.jinja --no-vision --draft-policy off \
  -c 65536 --k-bits 6 --v-bits 6 --default-max-tokens 32768

The model is served by the same server as our Qwen packs; --chat-template points it at this model's own template, which is included in the pack. OpenAI-style HTTP API (/v1/chat/completions, streaming and not, /v1/models, /health). Flags:

• -c context in tokens. --k-bits / --v-bits bits of the KV cache. We recommend 6 / 6.
• --sessions N users served at once. The context is shared between them.
• --default-max-tokens reply budget when a request has no max_tokens, thinking included.

Hardware

Largest context that served a real long prompt, 6-bit KV cache: 77,824 tokens for one user (a 76K-token prompt, 93 s to the first token). 3 users: 73,728 tokens in total, stable over 3 rounds of 24K each. With a 4-bit cache, one user reaches 110,592 tokens, but we do not recommend it (see Quality).

For a server that runs for hours under agent load, leave a margin: in our agent runs, 2 users at 57,344 tokens ran clean, while 61,440 ran out of GPU memory after about 3 hours. The command above uses 65,536 for one user.

Quality (reference = the original BF16 model)

The reference is Nanbeige4.3-33B run from the original BF16 weights, layer by layer. KL measures how far the output distribution is from the reference (lower is closer). Top-1 is how often the most likely token is the same. 16-bit KV cache.

The code figure comes from a few outlier tokens. The encyclopedia text is the weakest domain.

KV cache quality

Teacher-forced, the last 1024 positions after 32K of real text, against the BF16 reference:

We recommend 6 / 6. A 4-bit cache clearly hurts this model, and 3 bit is not usable.

Evals

An agent client with shell and file tools, thinking on, the server's default sampling, one run per task, 6-bit KV cache. Qwen3.8-27B (EXL3 4.0 bpw) ran on the same card and engine, with its built-in draft head for speculative decoding. This pack ran without a draft model. The sets are small: one task more or less is noise.

Several of this pack's misses were tasks stopped by a wall-time limit (2 of the 30, 3 of the 14). Nanbeige thinks at length and decodes at half the speed of Qwen3.8-27B with its draft head on this card, so a task takes longer.

Speed

6-bit KV cache, greedy, real text, tokens per second.

Sampling

The server's defaults are temperature 1.0, top-p 0.95, top-k 20; a request can set its own. The upstream card recommends temperature 1.0 and top-p 0.95 for agent and tool use, and temperature 0.6 and top-p 0.95 for reasoning and chat. The model thinks at length: allow an output budget of at least 32768 tokens, thinking included.

Limits

• The context on a 24 GB card is far below the model's native 524,288 tokens: every layer has full attention, so the KV cache grows fast.
• Tool calls have no grammar constraint in the Kyojin server. A malformed call is possible; check it before you run it.
• With several users at once, an answer can differ from the same request run alone, at positions where two tokens are nearly tied.
• The 6/6 KV cache is lossy (KL 0.054 against 0.042 for 8/8 at 32K).
• The 3090 runs power-capped; speed on another card or power limit differs.
• This is not a quantisation validated by the model's authors. Test generated code and review consequential changes before you apply them.

Licence and credits

Apache 2.0. Base model: Nanbeige4.3-33B by Nanbeige, Nanbeige/Nanbeige4.3-33B, Apache 2.0. Modification: the weights were quantised and converted to the EXL3 format by Yamz Labs; the architecture, tokenizer and chat template are unchanged. Our additions: Copyright (c) 2026 Yamz Labs, Apache 2.0. Provided as is, without warranty.

Base model facts: 33.1B parameters, dense Llama architecture; 52 layers, all with full attention; 56 query heads and 8 key/value heads of dimension 128; hidden size 7168; vocabulary of 166,912 tokens; 524,288 tokens of context.

Engine: Kyojin, built on ExLlamaV3 by turboderp (MIT). Quantisation and packaging by Yamz Labs. This release is unofficial. Not affiliated with Nanbeige or turboderp. "Nanbeige" is a name of its authors.