Ling-3.0-flash — NVFP4, single GPU, MTP working, SwiGLU clamp fixed
Ling-3.0-flash quantized to NVFP4 so it runs on one 128GB-class GPU under SGLang, with
NEXTN multi-token prediction working and the SwiGLU clamp actually applied.
Upstream ships a 4×141GB reference deployment. This runs on one card.
Two things make this different from a plain conversion, and I explain both below so you can
reproduce them rather than trust me:
1. The MTP layer is preserved and correctly declared, so speculative decoding works.
2. SGLang silently drops Ling's SwiGLU clamp on the NVFP4 path. Without the fix in
sglang_patch/, this model — or any NVFP4 build of it — corrupts roughly two thirds of the
code it writes. The weights are fine. The serving stack is what needs patching.
⚠️ Read this first: apply sglang_patch/ or you get a broken model
Ling is trained with clamped SwiGLU on its late layers. The official config carries the
limits:
expert_swiglu_limit_list layers 35–41 = 4 (all other layers 0)
share_expert_swiglu_limit_list layers 34–39 = 5, layers 40–41 = 7SGLang reads those keys and implements the clamp — but the NVFP4 MoE scheme throws it away:
# sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4_moe.py
gemm1_clamp_limit=None, # hard-codedEvery other MoE runner in SGLang (triton, marlin, ascend, deep_gemm, mxint4) forwards the real
value. Only the NVFP4 scheme discards it. With no clamp, activations in the last seven layers can
run away, and a junk token wins a step:
stack.append(c)ulp
high = len(arr) - Advisory
elif arr[mid] < x Advisory:
seen.add(culp) # should be seen.add(c)It looks random and it recovers immediately, which is what makes it easy to misdiagnose as a bad
quantization. It is not. It is a missing clamp.
What I measured, before and after the patch
Detection is an AST check for identifiers that are loaded but never bound. Use that, not a
non-ASCII scan — the most common junk tokens (ulp, oly, culp, Advisory) are plain ASCII
and a charset filter sails straight past them.
| sampling | before patch | after patch |
|---|---:|---:|
| temp 0, thinking off — 15 tasks | 4/15 | 15/15 |
| temp 0, thinking off — 10 tasks | 2/10 | 10/10 |
| temp 0.6 / top_p 0.95 / top_k 20, thinking on (upstream recipe) | 0/10 | 10/10 |
| temp 0.6 / top_p 0.95 / top_k 20, thinking off | 1/10 | 10/10 |
| generated code executed, assertions checked | — | 8/8 |
The upstream-recommended sampling going from 0/10 to 10/10 is the clearest signal. Before the
patch that configuration was the worst one; after it, it is perfect.
Applying it
cd /python/sglang/srt
patch -p1 < sglang_patch/01-flashinfer_cutlass.patch
patch -p1 < sglang_patch/02-compressed_tensors_w4a4_nvfp4_moe.patchOr bind-mount the two patched files over the image. Three edits total:
layers/moe/moe_runner/flashinfer_cutlass.py— add aswiglu_limitfield to
FlashInferCutlassMoeQuantInfo and pass it into the fused-MoE call. The CUTLASS kernel already
accepts swiglu_limit; the sibling mxfp4 quant info passes it and the NVFP4 one simply never
did. Both call the same flashinfer_cutlass_fused_moe.
.../compressed_tensors_w4a4_nvfp4_moe.py— build the clamp tensor from
moe_runner_config.gemm1_clamp_limit (Bailing sets that, not swiglu_limit) and pass it.
The same file's TRTLLM branch gets the hard-coded None fixed too.
Verify the clamp is live
The patch logs once per layer. You want to see this:
[swiglu-clamp-patch] cutlass nvfp4 limit=None <- layers 0–34
[swiglu-clamp-patch] cutlass nvfp4 limit=4 <- layers 35–41If you never see limit=4, the clamp is not being applied and your output will be corrupted.
Check that first before blaming anything else.
Do not use the TRTLLM MoE backend with this model
--moe-runner-backend flashinfer_trtllm cannot run Ling at all. Bailing never sets
routing_method_type, and once you supply it (RoutingMethodType.DeepSeekV3 — Ling's routing is
sigmoid + expert bias, n_group 8, topk_group 4, top_k 8) the routing kernel still refuses the
shape:
Routing kernel expects #experts per group <= warp size (32),
got 512 experts / 8 groups = 64 experts per group512 experts over 8 groups is 64 per group. That is a compiled CUDA limit, not a config problem.
Use the default CUTLASS backend — which is where the clamp fix lives anyway.
MTP (NEXTN) — how it works here and what to watch
Ling ships one MTP layer (num_nextn_predict_layers: 1, layer 42). It drafts tokens that the main
model then verifies in a single batched pass, so you pay one forward pass for several accepted
tokens.
Layer 42 is BF16 in this checkpoint — 0 packed tensors, 1552 plain. It must be excluded from
the quantization targets. config.json here puts re:.layers\.42\.. first in ignore.
This matters more than it sounds. If a blanket re:.\.experts\.. target claims layer 42's
experts are FP4 while the tensors are BF16, the draft head loads under the wrong scheme and drafts
garbage. You get:
accept len: 1.00, accept rate: 0.00and speculation becomes pure overhead — it gets worse the deeper you draft (I measured 0.94× at
k=1, 0.80× at k=2, 0.71× at k=3 in that broken state). **An acceptance rate of 0.00 means your
draft head is broken, not that MTP is a bad idea for this model.** Read the acceptance number
before you conclude anything about speculative decoding.
With layer 42 loading correctly, acceptance is 0.81–0.85 and k=2 is the optimum:
| configuration | tok/s | vs baseline | accept len | accept rate |
|---|---:|---:|---|---|
| speculation off | 22.60 | 1.000× | — | — |
| NEXTN k=1 | 40.19 | 1.778× | 1.85–1.95 | 0.85–0.95 |
| NEXTN k=2 | 45.1 | ~2.0× | 2.62–2.70 | 0.81–0.85 |
| NEXTN k=3 | 40.48 | 1.791× | 2.58–2.90 | 0.53–0.63 |
Deeper is not better. k=3 drafts more but accepts less, and the extra verify work eats the gain.
Speculation is not byte-identical
Under greedy, speculation-off is 5/5 self-reproducible, but NEXTN k=2 matches it byte-for-byte on
only 1/5 prompts. The differences I saw were paraphrases, not corruption — floating-point
non-associativity in the batched verify pass picking a different token at a near-tie.
With --enable-deterministic-inference that rises to 4/5 at 38.26 tok/s. That mode needs one more
edit: raise MIN_DEEPGEMM_DIM in sglang/srt/batch_invariant_ops/batch_invariant_ops.py from 16
to 64, because the guard is N >= MIN_DEEPGEMM_DIM and Ling's N=32 MoE router otherwise dies with
DeepGEMM failed for matrix shapes M=1, N=32, K=2560.
Quantization layout
| component | precision |
|---|---|
| routed experts (mlp.experts.*, layers 0–41) | NVFP4 — e2m1 weights, FP8-e4m3 block scales, tensor_group, group size 16 |
| attention q/k/v/o_proj, f_proj, g_proj, b_proj | BF16 |
| q_conv1d, k_conv1d, v_conv1d, o_norm, A_log, dt_bias | BF16 |
| MLA kv_a_proj_with_mqa, kv_b_proj, attention.dense | BF16 |
| shared experts, mlp.gate (router), dense MLP (layers 0–1) | BF16 |
| lm_head, word_embeddings, all norms | BF16 |
| layer 42 (NEXTN/MTP), all 1552 tensors | BF16 |
f_proj and g_proj are the KDA decay and output gates. I keep them at BF16 — a 4-bit gate feeds
an exponential decay whose error compounds along the sequence.
Weights 71.8 GB resident, 76 GB on disk, ~300 s to load.
input_global_scale tensors are included — do not strip them
SGLang's CompressedTensorsW4A4Nvfp4MoE allocates w13_input_global_scale and
w2_input_global_scale with torch.empty() and fills them only from the checkpoint. A
checkpoint without them loads with no error and no warning, leaves those parameters holding
uninitialized memory, and computes
g_alphas = (1 / w13_input_global_scale) * w13_weight_scale_2 from garbage. The result is NaN
logits and an output of nothing but ! characters.
This repo ships 61,440 of them, set to 1.0 (float32), one beside every expert
weight_global_scale. With dynamic: "local" activations the per-block scale is computed at
runtime and the kernel applies alpha = 1/(g_a·g_w), so g_a = 1 leaves s_a_block = amax/6,
comfortably inside FP8 e4m3 range.
These are uncalibrated. A calibrated activation scale may do better; I have not measured that.
Why W4A4 and not W4A16
SGLang implements exactly one NVFP4 MoE path — _is_fp4a4_nvfp4 →
CompressedTensorsW4A4Nvfp4MoE. There is no weight-only branch: get_moe_scheme falls through
_is_wNa16_group_channel (INT-typed only) to the w8a8 predicates and raises. A W4A16 checkpoint
(input_activations: null) hits
AttributeError: 'NoneType' object has no attribute 'num_bits' in _is_static_tensor_w8a8 —
those predicates dereference input_quant with no None guard.
So config.json declares input_activations (4-bit, dynamic: "local", tensor_group) to select
the W4A4 path. That is why the community W4A16 NVFP4 builds of this model will not serve on SGLang.
Serving
Requires an SGLang build with bailing_moe_v3 support, plus sglang_patch/.
python3 -m sglang.launch_server \
--model-path --trust-remote-code \
--host 0.0.0.0 --port 30013 \
--tp-size 1 --ep-size 1 \
--mem-fraction-static 0.75 \
--context-length 32768 \
--max-running-requests 1 \
--chunked-prefill-size 2048 \
--disable-shared-experts-fusion \
--tool-call-parser glm45 \
--speculative-algorithm NEXTN \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3--disable-shared-experts-fusion is required — the shared experts are BF16 while the fused MoE
expects FP4.
Do not pass --reasoning-parser. With one active, any request sending
enable_thinking: false comes back with content: "" and the answer stranded in
reasoning_content: the model emits no … wrapper, so the parser never sees a
terminator and files the whole output as reasoning. Agentic harnesses that disable thinking will
see empty replies.
Native context is 262144 (rope_scaling: null); 32768 above is just what I run.
Provenance
Quantized from the official BF16 release with llmcompressor 0.12.0.1 /
compressed_tensors 0.17.1 via model_free_ptq, scheme NVFP4A16.
Sequential oneshot was not usable — compressed_tensors' from_accelerate path asserts on
disk/meta tensors on this hardware. The Bailing MoE layout also requires the fused expert maps to
be emptied, with MLA kept out through the ignore list instead.
Counts after quantization: 61,440 quantized expert tensors; 16 MLA tensors and all 1,552 layer-42
tensors left unpacked at BF16.
Credit
All credit to InclusionAI for Ling-3.0-flash. The SwiGLU clamp behaviour was first documented for
the GGUF conversion path by raulvidis/Ling-3.0-flash-ROCmFP4-STRIX-MTP-GGUF; the fix here is the
equivalent for SGLang's NVFP4 CUTLASS path.
License
MIT, inherited from inclusionAI/Ling-3.0-flash.
Other public builds of this model
Compiled from Hugging Face repository metadata — file sizes, shipped files, quant variant as named by each repo. No third-party build was run or benchmarked here, so this table makes no speed or quality claim about any of them. It is here so you can see the size and format options at a glance and pick what fits your hardware.
| Repository | Largest model file | Variant | Ships | Downloads | Likes |
| --- | ---: | --- | --- | ---: | ---: |
| AtomicChat/Ling-3.0-flash-NVFP4 | 4.66 GiB | NVFP4 | safetensors | 1219 | 5 |
| r0b0tlab/Ling-3.0-flash-NVFP4 | 7.72 GiB | NVFP4 | safetensors | 904 | 2 |
| olka-fi/Ling-3.0-flash-NVFP4 | 7.72 GiB | NVFP4 | safetensors | 3391 | 3 |
| sakamakismile/Ling-3.0-flash-W4A4-NVFP4 | 9.32 GiB | NVFP4 | safetensors | 341 | 6 |
| raulvidis/Ling-3.0-flash-ROCmFP4-STRIX-MTP-GGUF | 41.47 GiB | STRIX | 2 model files | 2389 | 8 |
| AtomicChat/Ling-3.0-flash-NVFP4-GGUF | 41.55 GiB | NVFP4 | 4 model files | 4600 | 3 |
Base model: inclusionAI/Ling-3.0-flash. Generated from Hub metadata; download counts move over time.
Acknowledgements
Base model authors — see base_model in the metadata above; all model weights,
licences and capabilities are theirs. This repository contributes quantisation and
measurement only.
and NVIDIA's NVFP4 tooling** — the formats and runtimes these builds target.