Ling-3.0-flash-NVFP4
NVFP4 (4-bit) quantization of inclusionAI/Ling-3.0-flash (BailingMoeV3, 42-layer hybrid MoE + MTP) produced with NVIDIA ModelOpt using the DeepSeek-V4 experts-only recipe, served natively by the inclusionAI vllm-ling-v3 fork (FlashInfer-CUTLASS NVFP4 MoE).
- Size: 81.3 GB (24 shards) — 3.14x compression over the 255 GB BF16 source
- Quantization: weight-only NVFP4 (E2M1 packed 2-per-byte, E4M3 block scales, FP32 per-tensor scale),
activation_scheme: dynamic - Scope: the 61,440 routed-expert projections (layers 2–41) are NVFP4; attention (KDA + MLA), shared experts, routers, norms, embeddings, lm_head and the full MTP layer remain BF16
- Hardware: designed for NVIDIA GB10 / SM120/SM121-class GPUs
Quickstart
# vllm-ling-v3 fork @ 92c104112 (see the reproducibility repo for the container)
git clone --branch ling_3_0 https://github.com/inclusionAI/vllm-ling-v3
cd vllm-ling-v3 && git checkout 92c104112
VLLM_USE_PRECOMPILED=1 uv venv --python 3.12
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
uv pip install "flashinfer==0.6.15.post1" \
https://github.com/flashinfer-ai/flashinfer/releases/download/v0.6.15.post1/flashinfer_cubin-0.6.15.post1-py3-none-any.whl
vllm serve r0b0tlab/Ling-3.0-flash-NVFP4 \
--trust-remote-code --kv-cache-dtype fp8 --mamba-cache-mode align \
--max-model-len 24576 --enable-auto-tool-choice --tool-call-parser ling3 \
--reasoning-parser ling3 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
Two project-local fork patches are required on SM121 (Triton MLA decode smem and NVFP4 input-scale defaults) — see github.com/r0b0tlab/ling-3.0-flash-nvfp4 (patches/apply_patches.py).
Note on
chat_template.jinja: this checkpoint ships withenable_thinkingdefaulting to off (harness-compatible; matches the published benchmark rows). Passchat_template_kwargs: {"enable_thinking": true}per request for reasoning traces, or restore the upstream template frominclusionAI/Ling-3.0-flashfor the native thinking-on default.
Quality (r0b0bench core-subset, official harnesses)
| lane | score |
|---|---|
| GSM8K 0-shot (200) | 94.5 % |
| ARC-Easy test (400) | 96.0 % |
| IFEval (lightweight, 200) | 83.0 % |
| HumanEval pass@1 (164) | 65.9 % |
| BFCL v4 multi-turn base (200) | 62.0 % (76.5 % with thinking on) |
| BFCL v4 AST micro (600) | 30.7 % |
| NIAH long-context | PASS (all depths) |
| Canary | 5/5 |
Performance (1x GB10 / SM121, MTP scale 1, CUDA graphs)
- Decode: 33.5 tok/s C1 (21.4 base AR graphs, 13.8 base AR eager)
- MTP acceptance: 73.2 %, mean acceptance length 1.73
- FP8 KV, 24K context, mamba cache align
Full sweep + methodology: github.com/r0b0tlab/ling-3.0-flash-nvfp4
Reproducibility
- BF16 source:
inclusionAI/Ling-3.0-flash@eb04638493fe9eac066ede636090a9425272c481(SHA-256 manifest in the reproducibility repo) - Converter + container:
r0b0tlab/ling-3.0-flash-nvfp4(docker run ... convertreproduces this checkpoint) - Bench harness:
r0b0tlab/r0b0bench(results ledger entries for this model)
License
MIT (mirrors the source model). Weights derived from inclusionAI/Ling-3.0-flash (MIT, © inclusionAI) — see the upstream model card for any additional terms. Quantization and validation by r0b0tlab.