Xing4.0-29B-Intent-NVFP4

NVFP4 (W4A4) build of XingChen-AGI/Xing4.0-29B-Intent, a schema-guided intent classification and structured-extraction model (29B total, ~4B active; MLA + MoE + manifold-constrained hyper-connections). Built and measured on one NVIDIA GB10 (DGX Spark class, sm_121) with vLLM 0.32.0 and the xing4 vLLM plugin. The MTP head is kept, so speculative decoding works.

21.5 GiB of weights (from 58.1 GiB BF16). Same extraction accuracy as BF16 in our test (F1 60.43 and 59.93 on two runs vs 59.85), and the full 320-request extraction run finishes in 386 s vs 677 s for BF16 on the same box (1.75×); with MTP 2 in 265 s (2.55×) at F1 58.52.

Accuracy vs the BF16 original (same box, same vLLM, same 320 requests)

F1 is scored the way the base model card describes (per-dataset field-value micro-F1, macro average) on examples built with the dataset's own build_examples.py from the test split of Xing4.0-29B-Intent-oss, 40 per set (2 per task, English and Chinese), greedy, thinking off; calibration used the train split only. The BF16 score is consistent with the 58 overall on the base model card. PPL: 32 × 1024 wikitext-2 test tokens; the same windows through Hugging Face transformers in BF16 give 9.5579.

¹ ee_duee_finance_chinese scores ~0 for every build: in the released test split the schema the prompt supplies names the category and fields differently from the gold labels (schema 财经/交易-上市 / 上市企业, labels 公司上市 / 上市公司), and the model follows the schema. ² get_weather, three requests; the one that passes is the request with "skip_special_tokens": false (see Serving) — without it the <tool_call> tags are stripped before the parser sees them.

Why attention stays BF16. Quantizing the attention projections too (first two NVFP4 rows) costs 3 F1 points with native FP4 kernels and, on either kernel, makes the model loop on long event-argument lists (CASIE: 20–21 answers hit the length limit vs 12 for BF16). Keeping q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj and o_proj in BF16 removes that loss for 1.4 GB.

Speed on one GB10 (single stream)

Decode = median of 3 runs of 400 generated tokens; prefill = median of 3 distinct prompts (no prefix-cache reuse). ³ forced to 400 tokens past the end of a short JSON answer, which flatters speculative decoding; prose is the cleaner number. ⁴ wall time of the full extraction run at 4 concurrent requests, where measured.

Serving accuracy: vLLM plugin patch

On GB10 with vLLM 0.32.0 and plugin commit 00f9694, BF16 measured PPL 10.8468 vs 9.5579 for the Hugging Face transformers reference on the same tokens, and 57.84 F1 vs 59.85 with the patch. The difference traces to the tilelang mhc_pre call inside the served model (the kernel on its own matches vLLM's torch reference). xing4-vllm-plugin-mhc-fix.patch in this repo replaces that one call with the reference math in plain torch (fp32 mixes, the ±30 mhc_h_res_clamp from the model config, 20 Sinkhorn iterations); vLLM's compiler fuses it. With it: PPL 9.6617 (within 1.1 % of the reference). This NVFP4 build on the unpatched plugin: PPL 11.2248 vs 9.9701 patched; the fix costs speed — decode 34.78 → 31.93 tok/s, prefill 8K 4032 → 3113 tok/s. Measured on GB10 only; we have not checked other GPUs.

Serving

git clone https://github.com/XingChen-AGI/xing4-vllm-plugin && cd xing4-vllm-plugin
git checkout 00f969429546f3e8b96d68954438d04a3e79d572
git apply /path/to/xing4-vllm-plugin-mhc-fix.patch     # from this repo
cd .. && docker run --gpus all --network host --ipc host -v $PWD/xing4-vllm-plugin:/plugin \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_PLUGINS=xing4_0 --entrypoint bash vllm/vllm-openai:v0.32.0 -c \
  "cp -r /plugin /tmp/p && pip install --no-deps /tmp/p && vllm serve kingjones777/Xing4.0-29B-Intent-NVFP4 \
    --trust-remote-code --max-model-len 16384 --enable-auto-tool-choice --tool-call-parser xing4_0"

• Faster, slightly less exact: add --speculative-config '{"method":"mtp","num_speculative_tokens":2}' to use the model's MTP head. Measured: 320-request run 265 s instead of 386 s, F1 58.52 instead of 60.43/59.93 (the batched verify pass is not bit-identical to plain decoding; two identical runs without MTP differed by 0.50 F1). MTP 1: 297 s, F1 58.96.
• For extraction, send "chat_template_kwargs": {"enable_thinking": false} (the template thinks by default).
• For tool calls, also send "skip_special_tokens": false.
• The request format is documented with the XingIntent SDK.
• vLLM picked FlashInferCutlassNvFp4LinearKernel and the FLASHINFER_CUTLASS NvFp4 MoE backend (native FP4) on GB10.

This is real W4A4, not weight-only

tensors 30555 quantized Linear 7416 census {'input_global_scale': 7416, 'weight_global_scale': 7416, 'weight_scale': 7416, 'weight_packed': 7416} fused groups 2472 MTP 212 weights 23.06 GB activations local

format: nvfp4-pack-quantized      quant_method: compressed-tensors
weights:            4-bit float, tensor_group, group_size 16
input_activations:  4-bit float, tensor_group, group_size 16, dynamic local

NVFP4: every routed and shared expert and the two dense MLPs. Original precision: all attention projections, embeddings, lm_head, every norm, the MoE routers and their FP32 e_score_correction_bias, all mHC parameters (FP32 hc_scale), and the whole MTP layer model.layers.40 (BF16, copied byte-for-byte from the source). Fused gate_proj + up_proj pairs share one global scale, as vLLM requires.

Provenance

llmcompressor 0.13.0 oneshot + QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head", "re:.*kv_b_proj$", "re:.*self_attn\\..*"]), compressed-tensors 0.18.0, transformers 5.14.1, torch 2.13.0. Calibration: 160 sequences — 128 Xing4.0-29B-Intent-oss train examples (prompt + reference answer through the model's chat template, ≤ 4096 tokens, never truncated) and 32 wikitext-2 train chunks of 2048 tokens. Every routed expert observed every calibration token (each expert runs on all tokens, then the layer returns the original routed result — checked bit-exact before calibrating). Source revision 070c34b10a7cb8dab87c9f602e38cd67826ab48f. Licence: Apache-2.0, inherited from the base model.