Nanbeige4.3-33B EXL3

Calibrated EXL3 quantizations of Nanbeige4.3-33B, revision c1093c501cc6460427c0ac634db7223d0450c0ee, under Apache-2.0. The main branch contains this README; model files are on the eight bitrate branches. Labels describe the decoder target bitrate. Every rung has a six-bit output head, unquantized embeddings and its own sensitivity allocation.

Quality and speed at a glance

All eight rungs have complete official held-out evidence, 298 required core trials and native default checks 4/4. Quality warnings are retained rather than dropping requested compression levels. The initial speed measurements below are Modal RTX PRO 6000 Server Edition, 600 W, nativeQ8 KV. 4.05 and 5.05 were measured on a Vega workstation instead (RTX PRO 6000 Workstation Edition, 580 W), with and without the DFlash2 drafter; see DFlash2.

The token-use chart is an off-policy prediction from teacher-forced stop mass; measured generation length for 4.05 is reported under Generated length and matched agentic control. Exact values for every chart are in the tables.

Low-rung warnings: 2.05 has pooled/agentic thinking-stop mass 0.929/0.939, both 95% CIs below the 0.97 quality floor. 2.55 has agentic stop mass 0.957 and coding-on accuracy difference -8.89 pp [95% CI -18.89, -1.11]. 3.05 has coding-on difference -11.11pp [-21.11, -3.33]. These rungs remain available but are not recommended for agentic/coding work. A smaller thinking-stop mass suggests longer thinking under an inverse-hazard proxy; it is not a measured generation-length increase.

4.05 Chinese caveat: original 37/46 versus BF16 43/46; fresh selected/alternative/BF16 each 84/90. The combined paired contrast remains -4.72 pp [-9.72, -0.28], with a narrow single-template and multiple-testing scope. Full replication details follow.

Which rung should I use?

These are capacity-planning recommendations from measured weight bytes and the model's exact KV geometry, rather than tests on each consumer GPU. FP16 KV costs 208 KiB/token: 16K is 3.25 GiB, 32K 6.5 GiB, 64K 13 GiB. Context means prompt plus generated tokens in the total cache pool, not just prompt length. The estimates assume one active sequence, chunked prefill and explicit FP16 KV; keep the listed remainder for CUDA/PyTorch, graphs and workspace. Reduce the pool if your environment needs more headroom. These estimates cover the target model; a drafter needs additional weights, cache and workspace.4.55 has better KLD than 4.05 but leaves only about 1.40 GiB at 16K on a 24 GiB GPU, so 4.05 is the safer useful-context recommendation. Q8 KV can extend capacity, but the FP16 quality evidence does not validate quantized-cache quality.

Calibration and held-out measurement

Converted with stock exllamav3 v1.6.4, commitda68d57de61781e5b74ad68698ac4caf36408d7a, native LDLQ/trellis, mul1 and automatic output scales. Each rung uses a native supported-rate tensor allocation optimized from 80% assistant thinking/answer/tool-call KLD plus 20% response-decision KLD, domain weighted. Nine complete measurement contexts contain 7,525 assistant positions, with 346 Hessian rows and one noise draw; this proxy is not a repeat-stability claim.

Calibration source: 1,213 complete Nanbeige BF16-weight/BF16-KV conversations, 4,054,730 tokens, rendered in its own template through actual EOS. Domain tokens: agentic coding/tool trajectories 34.96%, other code 14.98%, math 11.99%, chat 11.99%, Chinese 8.00%, multilingual 4.99%, structured 4.99%, long context 5.10%, STEM 3.00%. The measured-selected subset is 310 complete conversations / 1,044,043 tokens, max 28,366, seed 20261025. Manifest SHA256 4305272fa895081ac90038f656cf3eec2e40cedf17edc3110e459304abc12191; packed calibration SHA256 469eea04a01461763fcbe4710f352bae2793de6d692a685a4899ff46045adb56. Evaluation families are disjoint.

Headline KLD is KL(BF16||quant), nats, on 634,268 assistant positions including assistant history, with 1,523 assistant decisions. Eval-v1 has 272 complete conversations / 1,006,589 tokens; manifest SHA256 4fe31e47a0b7426e1e3f89fa4945343561681f5112ef125cd43958cac3556fe5. Quant scoring is native full-sequence uncached FP16 attention, no KV cache; reference is stored SGLang BF16-weight/BF16-KV EXTEND. The full padded 166,912 vocabulary is normalized. BF16 top 256 + aggregate tail is a coarsened-KLD lower bound; assistant-only full-vocabulary validation is retained per branch. Prompt/schema/user/tool-output positions are excluded from the headline. Generation uses native paged FP16 KV and is a distinct execution mode.

Stop fidelity

The recipe-selection quality band is [0.97, 1.03] for canonical pooled and agentic stop-mass point ratios. All requested rungs remain published with prominent deviations and core-regression warnings. 95% conversation-cluster CIs and the native-runtime BF16 control are reported alongside. Stop mass concerns current BF16 response positions; assistant history is excluded. Teacher-forced hazards do not establish generated length.

Generated length and matched agentic control

The selected4.05 expanded arm completed 500/500 trials with 0 infrastructure errors. At an8,192-token restriction, the pooled length ratio is partially identified in[0.978725, 1.045298], with95% bootstrap bound envelope[0.945861, 1.095004]. Cap hits were 79/500 versus BF16 84/500; risk ratio 0.940476 [0.847059, 1.037500]. There were 16 quant time-budget censors versus 4 BF16 early censors. The native quant mixed queue used concurrency32, whereas retained BF16 agentic controls used4. This walltime/load mismatch prevents attributing the difference to longer thinking. All500 original rows remain retained. Restricted mean concerns min(completion length,8192), not unrestricted completion length. These results do not establish ±3% length equivalence.

The separate matched agentic rerun is complete: 30 fixed coding-agent tasks ×2 seeds, 60 valid pairs, concurrency4 in both arms, 8,192 output tokens, 12 turns and 300 seconds per task. Existing BF16 controls were retained after fixture/parser/protocol verification. Native generation used paged FP16 KV. At 8,192 tokens, the restricted-mean length ratio has identification bounds [0.977195, 1.040305] and a 95% paired prompt-cluster bootstrap bound envelope [0.793257, 1.277307] (4,000 draws). Early censors are 1/60 quant versus 4/60 BF16. Output-cap hits are 2/60 versus 1/60; rate-difference 95% CI [-5.00, +10.00] percentage points. The cap-hit risk-ratio confidence interval has an unbounded upper endpoint. This replaces the concurrency-mismatched agentic comparison for interpretation, while preserving all original rows separately. It does not establish unrestricted-length or statistical equivalence.

Matched hidden-test accuracy is 58/60 for EXL3 and 58/60 for BF16. The paired difference is +0.00 percentage points [95% prompt-cluster CI -6.67, +6.67]. These are existing short30 hidden-test grades, summarized without new inference or rerunning tests; source/protocol SHA256 values are retained in EVAL_METRICS.json. Equal observed scores do not establish accuracy equivalence.

Chinese arithmetic replication, selected EXL34.05

This probe contains 30 numeric variants of one Chinese arithmetic template (20 with thinking on, 10 off), rather than broad Chinese language coverage. Sampling and prompts were identical across arms: T=0.6, top_p=0.95, 1,536-token cap. The fresh run used three new seeds (20261201/02/03), 90 trials per arm. All 360 fresh trials completed without infrastructure errors.

The selected recipe's original nine failures and canonical BF16's three failures were wrong numeric answers, with no language-switch, format/grader-parse, truncation or thinking-only failures. Selected failures covered seven prompts; one prompt failed all three original seeds. Detailed per-prompt counts and failed answers are retained in CHINESE_REPLICATE_AUDIT.md.

Accuracy differences below average the paired seed differences within each prompt, then weight the 30 prompts equally. The 95% confidence intervals use 4,000 paired prompt-cluster bootstrap draws; repeated seeds are not independent prompt samples.

The fresh run did not reproduce the original gap. The combined canonical comparison remains marginally negative, while the same-runtime control comparison includes zero. Neither establishes equivalence. Around 30 artifacts by eight suites were examined: these nominal confidence intervals are unadjusted and do not control family-wise error across exploratory comparisons. The fresh contrasts were fixed before observing their results.

After this replication the selected recipe was retained. Native exllamav3 generation uses its pinned FP16 loading/cache path and native RNG; canonical SGLang uses BF16 weights/cache. Exact source hashes, runtime configurations, all results and the dated ruling are preserved in the evaluation scoreboard and CHINESE_REPLICATE_AUDIT.json.

Combined raw selected/canonical totals are 121/136 versus 127/136. Prompt-weighted paired estimates differ from pooled trial proportions because prompts receive equal weight. These probes do not establish broad Chinese equivalence.

Stock runtime compatibility

Every rung loads and decodes on unpatched v1.6.4 and v1.6.5, with identical cold/warm greedy outputs within each version. Cross-version token identity is not guaranteed. Full-vocabulary cross-version KL is at most 1% of the same fixed-bank quant-vs-BF16 KL on every rung. The control uses 20 positions on three short prompts and an actual BF16 Transformers4.57.6/Torch2.10 SDPA/noKV reference, separate from the official SGLang held-out scores. Scoring is uncached/noKV; greedy is native paged FP16 KV. Emitted IDs match the captured argmax. The Chinese 4.05 difference is 8/14 ULPs, not a one-ULP tie, with cross-version KL about 8e-6, far below quantization error.

Install and generate

Both stock releases were tested: v1.6.4 (da68d57de61781e5b74ad68698ac4caf36408d7a) and v1.6.5 (cdc785db89e263e65e5849cb53e1ada156411414), with Python 3.12, PyTorch 2.10.0 and CUDA 12.8. Choose either matching wheel:

python3.12 -m venv exl3-venv
exl3-venv/bin/pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
EXL3_VERSION=1.6.4  # 1.6.5 is also tested
exl3-venv/bin/pip install "https://github.com/turboderp-org/exllamav3/releases/download/v${EXL3_VERSION}/exllamav3-${EXL3_VERSION}+cu128.torch2.10.0-cp312-cp312-linux_x86_64.whl" transformers==4.57.6 sentencepiece==0.2.1 protobuf==6.33.5 huggingface_hub

This batch-one example uses the recommended 4.05 branch with a 16K total FP16 KV pool. Prompt plus generated tokens must fit the pool.

from huggingface_hub import snapshot_download
from exllamav3 import Cache, CacheLayer_fp16, Config, Generator, Model, Tokenizer, TopPSampler
from transformers import PreTrainedTokenizerFast

directory = snapshot_download("ProCreations/Nanbeige4.3-33B-EXL3", revision="4.05bpw")
config = Config.from_directory(directory)
config.override_dynamic_seq_len(2048)  # Chunked prefill workspace.
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=16384, layer_type=CacheLayer_fp16)
model.load(device="cuda:0", max_batch_size=1)
tokenizer = Tokenizer.from_config(config)
hf = PreTrainedTokenizerFast.from_pretrained(directory, local_files_only=True)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer, max_batch_size=1)
prompt = hf.apply_chat_template(
    [{"role": "user", "content": "Write a Python function that adds two integers."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
answer = generator.generate(
    prompt=prompt, max_new_tokens=512, stop_conditions=[166101],
    sampler=TopPSampler(top_p=0.95, temperature=0.6), seed=20261025,
    encode_special_tokens=True, add_bos=False, completion_only=True,
)
print(answer)

The tokenizer JSON is BPE despite the original LlamaTokenizer class declaration; explicit PreTrainedTokenizerFast avoids that ambiguity. Preserve EOS 166101, padded vocabulary 166912, the supplied chat template and RoPE theta 120000000. For agentic/tool trajectories, preserve thinking history, provide the tool schema and use the model's XML tool-call format. The source model recommends T=1.0/top_p=0.95 for agentic work and T=0.6/top_p=0.95 for chat/reasoning.

DFlash2

The Nanbeige4.3-33B-DFlash2 draft model has a native exllamav3 branch for these quants. It uses stock exllamav3 v1.6.4, and its card has a copy-paste quickstart that was run unmodified on the 4.05bpw branch. The draft shares this target's embedding and 6-bit head. exllamav3 samples the target and accepts matching draft tokens, so output follows the target's own sampling.

Single-stream decode on Vega, an RTX PRO 6000 Workstation Edition at 580 W: stock exllamav3 v1.6.4, Q8 target KV and FP16 draft KV, 25 prompts (five each from agentic, code, math, chat and Chinese), 256-token caps, plain and DFlash2 in the same process.

Every domain was faster than plain decoding on both branches at both temperatures. Per-domain rates and acceptance lengths are on the exllamav3 branch card. These short-prompt Vega rates use a different machine and protocol from the Modal table below, so don't compare the two tables row for row. Other rungs should work with the same draft but were not measured.

Measured speed on Modal

Modal RTX PRO 6000 Server Edition, 600 W; raw rates, no Vega normalization. Stock exllamav3 v1.6.4 (da68d57), Torch2.10/cu128, native internal graphs with warmed batch shapes, no source patch. Cache is native Q8 K8/V8, stored as packed int32 with FP16 scales. These speed results are distinct from official FP16-KV quality checks and do not validate Q8-cache quality. Full artifact-file hash checks passed before loading. All rates below are tokens/s.

Decode uses first-to-last token arrival; concurrency rates use total output / wall time including prefill. The 8K input/TTFT rate includes head, scheduling and host overhead and is not pure prefill-kernel throughput. The four lower-rung sweeps completed 18/19 cases, then reached the 780-second bound during concurrency128; that rate is missing and is not estimated. Their completed cases and native counts are verified. Rungs4.55/6.05 completed19/19. Exact raw samples, runtime/settings and manifest hashes remain with the serving exports. 4.05 and 5.05 were not run on Modal; their Vega measurements are in the DFlash2 section.

Files and integrity

Each model branch includes its own card, exact allocation, calibration/quality/compatibility evidence, LICENSE/NOTICE, FILELIST.json and MANIFEST.sha256. Weight, configuration and tokenizer hashes are unchanged from the tested artifacts. Internal run logs and workstation metadata are excluded from the public payload. The branch manifest lists every payload file and its SHA256; the manifest's own fingerprint is recorded by the release receipt.

Related releases: FP8, NVFP4-A16 and DFlash2. The base model's capabilities and limitations apply.

Frozen download sizes