Nanbeige4.3-33B NVFP4 W4A16

In multi-turn agentic tasks this quant runs noticeably longer than BF16 (about 12-17% more tokens, and more runs hit the 8K/12-turn limit). For agentic work, consider the EXL3 4.05-4.55 rungs or FP8.

NVFP4 weight-only quantization of Nanbeige4.3-33B, with BF16 activations, embeddings and untied output head. Stock SGLang 0.5.21 on SM120 / RTX PRO 6000 serves this format through Marlin. The checkpoint contains no embedded KV-cache scales: default auto uses BF16 KV, while explicit fp8_e4m3 uses unit scales.

On frozen eval-v1, assistant-token KL versus BF16 is 0.0329, decision KL 0.0174, and Chinese assistant KL 0.0496. Vega (our RTX PRO 6000 Blackwell workstation, 580 W cap) measures 70.03 tok/s single-stream with default BF16 KV. Thinking stop mass is slightly lower than BF16 on agentic histories; the measured shift and behaviour results are reported below.

Serving

python -m sglang.launch_server \
  --model-path ProCreations/Nanbeige4.3-33B-NVFP4-A16 \
  --dtype bfloat16 \
  --fp4-gemm-backend marlin \
  --reasoning-parser nanbeige \
  --tool-call-parser nanbeige

The stock automatic FP4 backend selects Marlin for W4A16_NVFP4 on this GPU; the command makes that choice explicit. Leave --kv-cache-dtype at auto for BF16 KV, or add --kv-cache-dtype fp8_e4m3 for a smaller unit-scale cache. Both settings have stock SM120 load/generation evidence. Independent default-auto smoke passed 4/4: thinking on/off, three weather tool calls and 8,259-token retrieval. Explicit-FP8 raw smoke passed arithmetic, coding, JSON, Chinese and thinking checks. Raw generation smoke is separate from task-accuracy evaluation.

Use the original tokenizer and chat template, including enable_thinking and preserve_thinking. Upstream sampling guidance is temperature 1.0 / top_p 0.95 for agentic/tool work and 0.6 / 0.95 for reasoning/chat.

In SGLang 0.5.21, the model-specific nanbeige parser names alias the Qwen3 reasoning and Qwen3Coder tool detectors. The shared OpenAI-endpoint contract was audited on this project’s FP8 checkpoint with streaming/non-streaming and both aliases; it was not repeated as an A16-specific parser audit. A literal </parameter> inside a tool string can silently truncate structured arguments while raw text stays complete. Validate arguments containing arbitrary code. The shared audit records 44/52 checks passed, with all eight failures attributable to this delimiter limitation. No SGLang patch is applied. See SERVING_PARSERS.md.

Quality versus BF16

Fixed-token assistant and decision KL

Frozen eval-v1 contains 634,268 assistant positions. Thinking, answers, tool calls and assistant history contribute; prompt/system/tool-output tokens remain in context and are excluded from the headline. Measurement uses SGLang native EXTEND/prefill, default-auto/BF16 KV, natural-log KL(BF16 ∥ quant), BF16 top-256 plus one complement bucket. This coarse KL is a lower bound on full-vocabulary KL. Decision labels include think-close, EOS, tool-close, answer-start and annotated reverification starts.

Behaviour suites versus BF16

All 298 required core trials are complete. No coding/tools/math regression has a paired prompt-bootstrap CI excluding zero under this release's rule (a coding/tool/math regression whose paired CI excludes zero would block release). agentic_coding_on is borderline: −7.8 pp with CI upper bound 0.0. This does not establish equivalence.

Small fixed-seed suites. Paired estimates average seeds within prompts and weight prompts equally; they need not equal differences between pooled trial counts. Confidence intervals cluster prompts. The independent expanded 500-pair length control is complete and reported below.

Thinking stop mass and generated lengths

Exact full-vocabulary think-close probabilities are measured at fresh BF16 thinking positions, excluding assistant history. These are teacher-forced native EXTEND/prefill indicators, not on-policy generated-length predictions. Intervals are 95% conversation-cluster bootstrap CIs.

Agentic think-close mass is about 1.3% lower on these fixed histories; its CI lies below 1.0. Both overall and agentic point ratios pass the revised [0.97, 1.03] envelope. The earlier stricter near-unity CI test is an advisory, not the current eligibility rule. Stop shifts remain disclosed rather than described as lossless stopping.

Expanded 500-pair length control

Complete 500/500 fixed pairs across 470 prompts, zero infrastructure errors, native SGLang 0.5.21 / SM120, default-auto/BF16 KV. Request-concurrency ceilings were 32 single-turn and 4 agentic, matching canonical BF16. Prompts, seeds, caps and agent protocol match the reference.

Agentic output-cap hits: 7/60 versus BF16 1/60 (+10 pp, 95% prompt-cluster CI [+3.3, +20]). Agentic restricted-mean length identification bounds are [1.118, 1.172], with 95% envelope [0.909, 1.486]. Overall bounds are [1.019, 1.024], 95% envelope [0.983, 1.063]; overall cap hits are 86 versus 84.

Restricted means concern min(completion length, restriction), not unrestricted completion lengths. Early time/turn stops create identification bounds; no independent-censoring assumption is made. The overall uncertainty interval includes 1.0. The agentic cap increase is significant in this fixed cohort; it is not a proven kernel mechanism or a general-task estimate.

All capped and early-censored rows are retained. Of the seven capped agentic trials, four passed hidden tests, three failed and one had a loop flag. Non-agentic completion checks in this length control are not answer-accuracy grades; the separate core table above reports task accuracy. No unrestricted completion-length or ±3% equivalence is claimed. See LENGTH_500.md and LENGTH_500.json.

Native decode and prefill geometry

Twenty real prompts, five per agentic/code/math/Chinese domain, generated greedily up to 256 tokens with EOS respected. BF16 rescored each exact generated sequence; the first generated token is excluded because it comes from prefill. No hooks, speculation or snapshots were used. The diagnostic uses common native decode/prefill top-20 support plus a complement bucket, with exact BF16 mass on that partition. Different artifacts produce different continuations, so these means do not rank candidates across artifacts and do not measure thinking-stop fidelity.

This control covers 4,684 decode positions. It is distinct from eval-v1’s fixed-token assistant estimator. No bit-exact equality across batching, graph widths or kernel geometries is claimed.

Speed on Vega: existing matched measurement

Stock SGLang 0.5.21 / SM120, FlashInfer, original BF16 head and embeddings, 2,048-token prefill chunks, CUDA decode graphs through batch 128, radix disabled and memory fraction 0.88. Random-token hardware prompts measure speed rather than reasoning quality.

All 19 exact hardware-bank cases passed for auto/BF16 KV. Mean/median timing summaries report 8K TTFT 1.697 s, with input-tokens/TTFT about 4,830 tok/s including first-token work. The long-context FP8 phase covered exact 131,072/256 and 250,000/128 input/output cases; it improves decode throughput but does not establish long-context FP8-cache quality. Relative to the same clean checkpoint with BF16 KV, unit-scale FP8 KV improves decode by about 30% at 128K and 44% at 250K. TTFT/prefill is not improved by those decode gains. Unit-scale and separately calibrated-KV companion speed were effectively equal in that phase. This repo is the clean unit-scale-capable checkpoint.

Auto/BF16-KV speed receipt: ba1f8e14ea81362c7e4b580b81454bdd7498d872a7f5a92a552863ceceb65d74. Long-context FP8 receipt: 688caf4b679cc709deb8eeba3026ca9f86765ad7660de28aa1079e672b0be994. See VEGA_SPEED.md and VEGA_FP8_LONG.md. The native 20-prompt referee uses different prompts and settings and is not a speed comparison.

DFlash2 v1 with this A16 target: Modal 600 W

A separate real-prompt benchmark on Modal RTX PRO 6000 Server Edition (600 W) measured the accepted DFlash2 v1 broad-retrain checkpoint with this exact A16 target. These are measured Modal rates; they are not normalized to the Vega workstation’s 580 W results above.

Protocol: 25 real prompts, five each agentic/code/chat/math/Chinese; full-bank warmup, EOS respected and a 256-token output cap. Stock SGLang 0.5.21, explicit --dtype bfloat16, BF16 target/draft KV, batch1 decode graphs, 2,048-token prefix chunks, fixed block8 (anchor plus seven proposals), one parallel draft pass and native stochastic conditional-q rejection. The drafter was trained with an FP8 teacher and was not A16-adapted for this measurement.

This is performance evidence, separate from task-quality parity. Only 7/25 greedy prompts were token-identical between plain and speculative runs; stochastic rejection correctness does not imply bit-exact floating-point paths. No accuracy or generated-length equivalence is inferred from this speed test.

The companion draft model is Nanbeige4.3-33B-DFlash2. See DFLASH_A16_SPEED.json for the measured rows and immutable identities.

Recipe, calibration and export

Base revision: c1093c501cc6460427c0ac634db7223d0450c0ee. This A16 checkpoint was independently calibrated with the weight-only NVFP4A16 scheme. FP4 E2M1 weights use groups of 16 with E4M3 block scales and a global weight scale; activations, embeddings and output head remain BF16. It does not reuse W4A4 activation quantization.

GPTQ uses block 128, dampening 0.01 and static activation ordering. Importance-weighted MSE scale search uses imatrix_mse, grid 40, squared-error norm 2, max-shrink 0.5, patience 5 and strict importance statistics. Export converts compressed-tensors names/global-scale reciprocals to SGLang’s modelopt / W4A16_NVFP4 loader convention, preserving packed FP4 values and block scales. This is a serialization conversion, not a claim that NVIDIA ModelOpt performed quantization.

Quantizer environment: llm-compressor 0.13.0, compressed-tensors 0.18.0, Transformers 5.12.1, PyTorch 2.13.0+cu130. Serving uses the stock image lmsysorg/sglang:v0.5.21, SGLang commit e00930c5489053f26d86b179cee0d087f846acbb, FlashInfer 0.6.18 and sglang-kernel 0.4.7.

Frozen calib-v1 contains 1,213 EOS-complete BF16-weight/BF16-KV conversations and 4,054,730 tokens. Source token mix is approximately 35% agentic coding, 15% other code, 12% math, 12% chat, 8% Chinese, 5% multilingual, 5% structured/tool, 5% long context and 3% STEM. Selected calibration uses 556 native windows / 1,086,255 tokens, seed 43, normally 2K with retained long windows up to 32K. No eval data is used for calibration or sensitivity allocation.

Calibration manifest SHA256: 4305272fa895081ac90038f656cf3eec2e40cedf17edc3110e459304abc12191.

Eval-v1 manifest SHA256: 4fe31e47a0b7426e1e3f89fa4945343561681f5112ef125cd43958cac3556fe5.

Quantization took 3,140.29 s on Modal RTX PRO 6000; the A16 format export took 29.64 s. The clean copy removes all 104 KV-scale tensors and the KV scheme, taking 135.36 s on CPU while preserving weights and linear scales. Tensor bytes: 22,070,230,448. See QUANT_CARD.md.

Compatibility and limits

• Stock SGLang 0.5.21 SM120 Marlin load, graph and native generation are verified for this weight-only format. vLLM and Transformers execution are not runtime-tested.
• This clean checkpoint has no embedded KV scales. Separate embedded non-unit FP8-KV-scale checkpoints are unsafe with explicit BF16 KV in SGLang 0.5.21 FlashInfer: matching-dtype writes skip scale division while reads apply the loaded scales. That caveat does not imply stock auto is broken.
• Short explicit-FP8 smoke and hardware long-context speed are not a full FP8-cache long-context quality gate. Default BF16 KV is the quality-tested reference here.
• Small fixed suites and teacher-forced KL/stop mass do not establish general accuracy, generated-length or context-length equivalence. The expanded 500-pair control is complete, with the agentic cap increase disclosed above.

License and related models

Apache-2.0, inherited from the base model. See LICENSE. Weight precision and serialization change; the original BF16 head/embeddings remain.

The project's FP8 checkpoint and DFlash2 draft model are published; EXL3 is also published.