Ling-3.0-flash — NVFP4 W4A4 (calibrated, MTP verified working)

The first W4A4 NVFP4 quantization of Ling-3.0-flash

(124B total / 5.1B active, hybrid KDA+MLA MoE) — weights and activations in NVFP4,

calibrated with 128 samples via llm-compressor.

| | |

|---|---|

| Size | 81.4 GB (from 255 GB BF16) |

| Format | compressed-tensors nvfp4-pack-quantized, W4A4 (E2M1 + FP8-E4M3 group-16 scales, static input global scales) |

| Measured | 110 tok/s single / 493 tok/s @ 8-way generation, ~7,000 tok/s prefill — 8×RTX PRO 2000 Blackwell (16 GB), TP=8, 32K context |

| Long context | 131,072 tokens verified (eager mode; prefill 5,900–8,000 tok/s) |

| MTP | Works: 93% acceptance (k=1). Off by default — see below |

| Sanity | 6/6 (math / logic / code / JP idiom / instruction-following / long-form JP); tool-calling 4/4 |

Recommended: fp8 KV cache — 131K context at full speed

With the bundled vllm_patch/triton_decode_attention.py

(1-line shared-memory fix for sm_120), --kv-cache-dtype fp8 works and everything fits at once:

|---|---|---|---|---|

458K tokens of KV = three concurrent 131K streams, or eight at 57K each.

Quality battery under fp8 KV: 6/6 (math / logic / code / JP idiom / instruction-following /

long-form JP) — no measured degradation vs bf16 KV.

Single-stream prefill in the recommended config is chunking-bound (`--max-num-batched-tokens

2048`, the largest stable value at 131K on 16 GB); parallel prefill reaches ~6,500 tok/s.

Why context is cheap here but headroom is not. Only 7 of 42 layers are MLA (the other

35 are KDA linear attention and carry no KV), so the KV cost is 8.2 KB/token — roughly

9× cheaper per token than an all-MLA model like DeepSeek-V3. The constraint is the card:

weights + non-torch overhead take 12.66 GB of a 16 GB GPU, leaving ~1.5 GB for KV and

KDA state — a total budget of about 187K tokens to split between length and concurrency

(131K × 1 stream, or 23K × 8 streams, etc.). On 80 GB cards this limit essentially vanishes.

How this differs from olka-fi/Ling-3.0-flash-NVFP4

(weight-only W4A16): this build quantizes activations too (W4A4) with real calibration data,

and ships a working MTP path.

Quantization boundary

Same proven boundary as the W4A16 build: routed experts only (layers 2–41,

mlp.experts.*.{gate,up,down}_proj, 120.8B of 127.4B params) → NVFP4 W4A4.

Kept in BF16: shared experts, KDA & gated-MLA attention, router gate + expert_bias,

dense layers 0–1, the MTP layer 42, embeddings, lm_head, norms.

Serving (vLLM fork required)

Upstream vLLM does not support bailing_hybrid v3. Use the prebuilt image

olkafi/vllm-bailing-v3 (inclusionAI/vllm @ ling_3_0 + SwiGLU-clamp patch).

This exact command reproduces every number in this card (131K context, fp8 KV,

458,752-token KV budget, ~112 tok/s single-stream). Both bundled patch files are

mounted over the image's own copies — no rebuild needed, they are pure Python:

REPO=/path/to/this-repo   # e.g. $(huggingface-cli download sakamakismile/Ling-3.0-flash-W4A4-NVFP4)

docker run --gpus all -d --name ling3 --ipc=host -p 8000:8000 \
  -e NCCL_P2P_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 \
  -v $REPO:/models/ling3:ro \
  -v $REPO/vllm_patch/unquantized.py:/opt/vllm-ling3/vllm/model_executor/layers/fused_moe/oracle/unquantized.py:ro \
  -v $REPO/vllm_patch/triton_decode_attention.py:/opt/vllm-ling3/vllm/v1/attention/ops/triton_decode_attention.py:ro \
  olkafi/vllm-bailing-v3:latest /models/ling3 \
  --served-model-name ling3 --trust-remote-code --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 8 --disable-custom-all-reduce \
  --kernel-config '{"moe_backend":"marlin"}' --kv-cache-dtype fp8 \
  --compilation-config '{"max_cudagraph_capture_size":16}' \
  --gpu-memory-utilization 0.95 --max-model-len 131072 --max-num-seqs 8 \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching --mamba-cache-mode align \
  --enable-auto-tool-choice --tool-call-parser ling3 --reasoning-parser ling3

Notes for reproducers:

--max-num-batched-tokens 2048: 4096 OOMs at 131K on 16 GB cards (chunked-prefill

activations); 2048 is stable.

On cards with more than 16 GB, drop the two patch mounts if you don't need them

(flashinfer_cutlass may still be broken on sm_120 — keep moe_backend=marlin there),

and raise --max-num-batched-tokens / --max-num-seqs freely.

NCCL flags are for machines without GPU P2P; harmless otherwise.

Sampling (from the base card): temperature=0.6, top_p=0.95, top_k=20, thinking enabled.

⚠️ Known landmines (sm_120 / consumer Blackwell)

1. moe_backend=marlin is mandatory. The auto-selected flashinfer_cutlass

backend silently corrupts output (endless !!!!) on sm_120 with EP, and plain

TP=8 hits NotImplementedError (per-rank intermediate 768/8=96 needs unsupported

padding). cutlass rejects EP. Marlin works and is the source of the numbers above.

2. KV headroom is tight — on 16 GB cards use --gpu-memory-utilization 0.95,

--max-num-seqs 8 and --compilation-config '{"max_cudagraph_capture_size":16}'.

32K fits comfortably that way; 131K needs --enforce-eager (see table above).

3. --kv-cache-dtype fp8 needs the bundled kernel patch on sm_120. The TRITON_MLA

fp8 decode kernel asks for 102,400 bytes of shared memory at num_stages=2; consumer

Blackwell caps at 101,376 — 1 KB short. vllm_patch/triton_decode_attention.py

drops fp8-KV MLA to num_stages=1, which fits with no measured speed loss

(112.6 tok/s single-stream, same as bf16 KV).

4. The bundled vllm_patch/unquantized.py makes the

unquantized MTP-layer MoE fall back to triton when moe_backend=marlin is forced

globally — without it, serving with --speculative-config fails at startup.

MTP / speculative decoding: verified working, off by default

The W4A16 release reported 0% acceptance and shipped with MTP disabled. With this

build + the bundled patch, MTP works:

| config | single-stream | acceptance |

|---|---|---|

| no MTP (recommended) | 113.2 tok/s | — |

| MTP k=1 | 88.3 tok/s | 93.0% |

| MTP k=2 | 110.0 tok/s | 68%/tok |

The draft (BF16 MTP layer on the triton path) currently costs more than the accepted

tokens buy back on this hardware, so MTP is a proof-of-life, not a speed win — leave

--speculative-config off for throughput. To try it:

--speculative-config '{"method":"mtp","num_speculative_tokens":1}' (add

--compilation-config '{"max_cudagraph_capture_size":32}' --max-num-seqs 32 for memory).

Bake provenance

llm-compressor 0.12 (lna-lab/llmc-bake:v0.24.0), 128 calibration samples

(neuralmagic/calibration, chat-templated) @ 2048 tokens, basic pipeline with

CPU-resident weights and single-GPU onload calibration (fla/KDA kernels are

triton/cuda-only, so pure-CPU forward is impossible). Recipe quirks that will bite

reproducers: transformers 5.x needs three shims in the custom modeling file

(is_torch_fx_available, ROPE_INIT_FUNCTIONS['default'], _tied_weights_keys);

rope_scaling must be forced back to None after config normalization;

llm-compressor drops model_type from the saved config (breaking vLLM's MLA

detection — restore "model_type": "bailing_hybrid"); and post-calibration weight

observation must be moved to CPU (set_onload_device) or 61,440 expert observers

accumulate on one GPU and OOM.

Acknowledgements

Built on inclusionAI's Ling-3.0-flash (MIT). Quantization boundary and serving

groundwork follow olka-fi's pioneering W4A16 release and vendor-fork patches.

MIT license, inherited from the base model.