Nanbeige4.3-33B FP8

Versus BF16, assistant-token KL is 0.0074 and thinking stop-mass ratio is 1.000 [0.996, 1.004] (Chinese 0.987, slightly later think-close). Matched 30-task × 3-seed agentic coding is 82/90 vs 86/90 (−4.4 pp [−11.1, +2.2], inconclusive); long-context retrieval passes 6/6 at >100K. Vega (our RTX PRO 6000 Blackwell workstation, 580 W cap) delivers 44.2 tok/s single-stream with default BF16 KV, and this checkpoint is about 32% faster than primitive-ai FP8 at matched FP8 KV.

W8A8 FP8 quantization of Nanbeige4.3-33B: E4M3 weights in 128×128 blocks and dynamic per-token/group-128 activation scales. Embeddings and the untied output head remain BF16. The HF-format checkpoint is tested with stock SGLang 0.5.21 on SM120 / RTX PRO 6000.

This is the clean weights-only export: it contains no embedded KV-cache scales. Default --kv-cache-dtype auto uses BF16 KV; explicit fp8_e4m3 uses unit cache scales.

Serving

python -m sglang.launch_server \
  --model-path ProCreations/Nanbeige4.3-33B-FP8 \
  --dtype bfloat16 \
  --reasoning-parser nanbeige \
  --tool-call-parser nanbeige

The command leaves --kv-cache-dtype at its stock auto default. For a smaller cache, add --kv-cache-dtype fp8_e4m3. Both settings passed six raw native generation checks, including arithmetic, code, JSON, Chinese and thinking. The independent default-auto release smoke passed 4/4: thinking on/off, three weather tool calls, and 8,259-token retrieval. These runtime checks do not establish task accuracy or length equivalence. In SGLang 0.5.21, nanbeige names alias the Qwen3 reasoning and Qwen3Coder tool detectors. OpenAI-endpoint parser coverage and limitations are recorded separately from the raw-generation smoke.

The OpenAI /v1/chat/completions audit verified both nanbeige/nanbeige and qwen3/qwen3_coder, streaming and non-streaming, with same-generation raw sampled-token controls. Thinking on/off, empty thinking, single and parallel calls, nested JSON, preserved multi-turn thinking/tool history, and tool-choice auto/none pass. 44/52 checks passed: the eight failures are the same known XML delimiter limitation, reproduced with natural multiline code and exact-wire controls in both modes under both aliases. A literal </parameter> inside a string silently truncates structured arguments at that tag, while the raw text remains complete. No SGLang patch is applied. Validate tool arguments; arbitrary code strings containing this delimiter are not supported by the stock parser contract. See SERVING_PARSERS.md and its raw JSON audit.

Use the original tokenizer and chat template. enable_thinking controls thinking; preserve_thinking=True is useful for multi-turn agent/tool histories. Upstream sampling guidance is temperature 1.0 / top_p 0.95 for agentic/tool work, and 0.6 / 0.95 for reasoning/chat. Context capacity is hardware/cache dependent; the base model's configured limit is not a promise that every serving setup fits it.

Quality versus BF16

The native 20-prompt decode control measured KL 0.0043 overall, Chinese 0.0099. Frozen eval-v1 assistant-only prefill measured KL 0.007368, decision KL 0.004023, and top-1 agreement 97.82%. These are distinct corpora and estimators, not interchangeable measurements.

Behaviour suites versus BF16

Small fixed-seed suites; paired differences and CIs are prompt-clustered (not pooled trial-rate differences); no equivalence claimed.

Source: frozen eval/scoreboard.json, updated 2026-10-25 21:31:53 UTC.

Held-out assistant and decision KL

Frozen eval-v1 has 634,268 assistant positions, including thinking, answers, tool calls and assistant history. Prompt/system/tool-output positions stay in context but are excluded from this headline. SGLang native EXTEND/prefill, auto/BF16 KV; natural-log KL(BF16 ∥ quant), BF16 top-256 plus an aggregate complement. This coarse KL is a lower bound on full-vocabulary KL. Decision positions include think-close, EOS, tool-close, answer-start and annotated reverification starts.

Native autoregressive decode and the prefill gap

Twenty real prompts, five each agentic/code/math/Chinese; greedy generation up to 256 tokens, respecting EOS. Each generated continuation is independently rescored with BF16 weights/BF16 KV. The first generated token is excluded because it comes from prefill. No hooks, speculative decoding or snapshots are used. KL uses the shared decode/prefill top-20 intersection plus an aggregate complement, with exact BF16 mass on that partition. This lower-bound diagnostic covers 4,711 decode positions; it is not the full-v1 estimator. Different artifacts generate different continuations, limiting cross-artifact comparisons.

Stop fidelity and coding

Primary stop-fidelity evidence uses exact full-vocabulary log-softmax masses at fresh BF16 thinking positions in eval-v1, excluding assistant history, teacher-forced native EXTEND/prefill with auto/BF16 KV. Confidence intervals are 95% conversation-cluster bootstrap intervals, 4,000 draws.

• Pooled thinking stop-mass ratio versus BF16: 1.000123 [0.996135, 1.004320], over 395,963 positions / 252 conversations.
• Agentic thinking stop-mass ratio: 1.008823 [0.999439, 1.018957].
• Chinese thinking stop-mass ratio: 0.986585 [0.973874, 0.995896]. Chinese shows localized underweighting of think-close despite the pooled ratio being near 1.
• Agentic reverification first-token mass ratio: 1.017869 [1.010261, 1.025196]. This is a lexical Wait/wait/Let first-token proxy, not joint-phrase or semantic-loop probability.

Full 30-task, three-seed coding result: 82/90 vs BF16 86/90 (-4.4 pp, CI [-11.1, +2.2], inconclusive). The paired interval clusters prompts. Protocol total-token ratio was 1.04761 [0.91696, 1.21160]; unfinished trials were 6/90 versus BF16 7/90, and loop flags 5/90 versus 4/90. This fixed suite does not establish accuracy equivalence or ±3% length equivalence. Earlier eight-task pilot comparisons are superseded by this result. Lower KL and near-unity teacher-forced stop mass do not guarantee equal generated lengths or coding accuracy.

Speed on Vega

The exact public clean checkpoint with default auto, physically BF16 KV, delivers 44.20 tok/s at 1K input / 1K output (median of three cases). Vega RTX PRO 6000 SM120, stock SGLang 0.5.21, plain decoding without a draft model:

For clean auto/BF16 KV, 8K prefill input-tokens/TTFT is 7,420.86 tok/s (about 7.4K tok/s), median TTFT 1.104 s. This estimate includes first-token head, scheduler and HTTP work; it is not isolated GEMM throughput. On the 96 GB card at --mem-fraction-static 0.88, the BF16 KV pool actually holds 248,865 tokens, despite the configured 262,144 context. The maximum server prompt is 248,859 tokens; the 250,000-input benchmark was skipped, and the largest measured prompt is 131,072 tokens. Pool capacity is not a quality-tested context-length claim.

For the same FP8 cache dtype, the clean unit-FP8-KV row is about 32% faster single-stream than primitive-ai FP8 (44.08 vs 33.42 tok/s), and about 23% faster at 128K (31.26 vs 25.41 tok/s). The earlier 43.36 tok/s, about 30% above primitive FP8, belongs only to the separately labelled calibrated-KV companion.

Primitive's 1,811 tok/s aggregate is the initial project baseline with FP8 KV and an approximately 497,671-token pool, using 8,192-token prefill chunks. Its later matched 2,048-chunk bank measured 1,854.5 tok/s. The clean default row uses BF16 KV and a smaller pool; clean unit-FP8-KV concurrency was not measured. Primitive's higher aggregate came with FP8 KV and the larger cache pool, so it is not evidence of faster weights under matched cache conditions. These configuration differences prevent attributing the concurrency difference to weight format alone.

Context: above roughly 248K total context tokens on this one-card/0.88 setup, use --kv-cache-dtype fp8_e4m3. The uncapped FP8 pool is approximately 0.5M tokens, estimated from the BF16 pool and half-sized KV elements and consistent with the measured primitive FP8 pool. The optional clean FP8 benchmark explicitly capped its pool at 278,528 tokens. The full 512K (524,288-token) base-model context still does not fit on one 96 GB card at 0.88 with this checkpoint; cache capacity also has to cover generated tokens and runtime overhead.

Clean hardware coverage: all 18 auto/BF16-KV and two unit-FP8-KV admitted cases passed exact token counts; the latter phase took 130.89 s including startup/cleanup. FlashInfer, stock image b1259f3, 128 scheduler/graph slots, chunk size 2,048, radix cache disabled, mem_fraction_static 0.88, original BF16 head/embeddings. Canonical prompt-bank SHA256 305ebb7470fb1ca0a9277c98b2daf2330f4c57a0428b5f4df5f30085dcab91db; raw export receipt dad8f0cfdc609b357ffdc8d274f3c5ccb89a17ab6b8ba507242e6da1aca19cb9. See VEGA_SPEED.md and its JSON summary. The random-token hardware bank measures speed, not reasoning, accuracy or length fidelity.

Recipe and calibration

Base revision: c1093c501cc6460427c0ac634db7223d0450c0ee. Quantizer: llm-compressor 0.13.0, compressed-tensors 0.18.0, Transformers 5.12.1, PyTorch 2.13.0+cu130. Scheme FP8_BLOCK: per-128×128 E4M3 weight scales, dynamic per-token/group-128 activation scales; output head and embeddings BF16. Standard minmax weight quantization is data independent. Calibration supplies activation/KV observations for the original companion; the clean export removes all 104 KV-scale tensors and the KV scheme without altering linear weights/scales. This is not a GPTQ FP8 artifact.

Frozen calib-v1 contains 1,213 EOS-complete BF16-weight/BF16-KV conversations, 4,054,730 tokens. Source token mix: approximately 35% agentic coding, 15% other code, 12% math, 12% chat, 8% Chinese, 5% multilingual, 5% structured/tool, 5% long context, 3% STEM. Selected calibration: 556 windows / 1,086,255 native token IDs, seed 43, normally 2K with retained long windows up to 32K. All nine absolute domain budgets were met; long-window rounding gives a selected long-context share of 7.41%. Decision tails and actual EOS are preserved where present. No eval data was used for calibration or selection.

Calibration manifest SHA256: 4305272fa895081ac90038f656cf3eec2e40cedf17edc3110e459304abc12191.

Eval-v1 manifest SHA256: 4fe31e47a0b7426e1e3f89fa4945343561681f5112ef125cd43958cac3556fe5.

Calibration/export took 579.44 s on Modal RTX PRO 6000; removing KV metadata/tensors for the clean copy took 262.58 s on CPU. See QUANT_CARD.md for cost, file identity and additional runtime provenance.

Compatibility and limitations

• Tested serving image: lmsysorg/sglang:v0.5.21, SGLang commit e00930c5489053f26d86b179cee0d087f846acbb; FlashInfer 0.6.18, sglang-kernel 0.4.7, CUDA 13.0. Stock loaders/kernels are used.
• An embedded non-unit FP8-KV-scale checkpoint combined with explicit BF16 KV is unsafe in SGLang 0.5.21 FlashInfer: BF16 writes skip scale division while decode reads apply the loaded scales. This clean checkpoint has no embedded scales and avoids that combination. Stock auto passed for both clean and calibrated companions; the explicit-BF16 failure does not imply an auto-default failure.
• Calibrated-KV companions remain separate diagnostic artifacts. Their long-context/cache quality cannot be inferred from prefill KL alone. No calibrated scales are silently included here.
• vLLM and Transformers execution are not runtime-tested for this release. HF-format serialization alone is not a compatibility claim.
• Native/pre-fill KL and exact stop bias measure different execution paths. No bit-exact output equivalence across batching, graph width or kernel geometry is claimed. Full general-task and long-context quality coverage remains limited; all planned suites for this artifact are complete; coverage is limited to the suites reported above.

License and related models

Apache-2.0, inherited from the base model. See LICENSE. This repository changes weight precision and cache metadata, with evaluation and quantization documentation added.

Other Nanbeige4.3-33B formats from this project: NVFP4-A16, EXL3, 2.05-6.05 bpw branches and DFlash2 draft model.