Ling-3.0-flash-VL-NVFP4-MP
Max-performance mixed NVFP4 + FP8 quantization of
inclusionAI/Ling-3.0-flash-VL
(124 B total / 5.1 B active hybrid KDA + MLA MoE vision-language model, MIT), produced with
NVIDIA Model Optimizer 0.48 and served by the
inclusionAI vllm-ling-v3 fork
(e2e5751 + two patches) in a purpose-built SM121 container.
- Size: 68 GB (64 shards) — 3.7× compression over the 249.7 GB BF16 source
- Runtime:
ghcr.io/r0b0tlab/ling30vl-nvfp4-mp-sm121:e2e5751(arm64, SM12x) - Repro package (click-run): r0b0tlab/ling30vl-nvfp4-mp-sm121
- Hardware validated: single NVIDIA GB10 / SM 12.1 (121 GB unified)
Quantization
| scope | modules | format |
|---|---|---|
| routed experts (layers 2–41, gate/up/down) | 61,440 | NVFP4 W4A4, group 16, E4M3 block scales, FP32 global amax/(6·448), dynamic per-token activations, input_scale = 1.0 |
| shared experts | 120 | NVFP4 (same) |
| dense MLP (layers 0–1) | 6 | NVFP4 (same) |
lm_head |
1 | NVFP4 (same) |
| attention (KDA q/k/v/b/f/g/o + MLA q/g/kv_a/kv_b/dense) | 280 | FP8 per-channel weight, per-token dynamic activation |
vision tower, projector, embeddings, norms, conv1d, routers, A_log, dt_bias |
716 | BF16/FP32 preserved (byte-identical) |
Audits: G0 census 16/16 · canary NVFP4 cos ≥ 0.9950 / FP8 cos ≥ 0.9997 · metadata contract · coverage 61,847/61,847 modules · G3 audit PASS (186,104 index keys).
Size, memory, and KV cache (single GB10, 121 GiB unified)
- Artifact: 72.62 GB / 67.64 GiB (64 shards; 72,597,622,664 bytes of weights + 26.6 MB metadata) — 3.44× smaller than the 249.7 GB BF16 source.
- GPU memory at the winner config (
--gpu-memory-utilization 0.85): weights 67.97 GiB + CUDA graph capture ~1.0 GiB + KV pool 30.49 GiB ≈ 99.5 of 121 GiB unified. - KV cache (fp8): 4,032 B/token (7 MLA layers × 576 B); 2,309,051 tokens capacity → 140.9× concurrency at 16 K, 70.4× at 32 K, 17.6× at the 131,072-token advertised window. fp8 is the maximum — NVFP4 KV is unavailable for this dense-MLA model on SM121.
Evaluation (2026-09-09, single GB10)
Q200v2 text180 (frozen quality-text-180-v2, thinking on, max_tokens 8192):
gsm8k 78/80 · humaneval 40/40 · ifeval 38/40 · hard_reasoning 20/20
(independent manual review) → 176/179 transported = 98.3 %.
One row exceeded the 8192-token ceiling (finish_reason=length) and is excluded by the
frozen contract (its grader passed) — disclosed, not hidden.
End-to-end throughput 42.64 tok/s mean / 43.13 aggregate (n=180); telemetry load
38.05 W mean, 68.3 °C mean, 5.07 GiB min MemAvailable.
BFCL v4 multi_turn_base structural-hard20 (official bfcl_eval 2025.12.17 checker):
17/20 = 85.0 %, 20/20 rows, 0 errors.
Q200v2 combined: 193/199 transported (97.0 %).
NIAH — advertised window (25/50/90 % depth + multi-key 33/66): 15/15 across 32 K / 65 K / 131 K ladders, 0 infra errors.
r0b0bench-vision v1.0 (4 suites, 4,703 rows, thinking-off): 3,801/4,703 = 80.82 % — CV-Bench 78.9 % (Count 67.1 · Relation 84.5 · Depth 92.8 · Distance 74.5), MMVP 82.0 % (paired 66.7 %), RealWorldQA 78.3 %, OCRBench 874/1,000.
Quality (single GB10, this checkpoint + container)
| gate | result |
|---|---|
| text probes | 5/5 (17×23 = 391, fib(10) = 55, strict JSON, code, real tool_calls) |
| vision probes | 8/8 (counting ×4, 7-segment OCR, bar chart, frame order/colour) |
| GSM8K-200 (flexible extract) | 91.5 % (183/200) |
Performance (single GB10 / SM121, c = 1)
| metric | value |
|---|---|
| decode | 43.5 tok/s |
| prefill | 986 tok/s (TTFT 13.0 s on 12,832 tokens) |
| ITL | 22.6 ms |
| aggregate | 42.4 → 164.8 → 165.4 tok/s at c = 1 / 16 / 32 |
| load / KV | 516–540 s, 67.97 GiB / 31.8 GiB → 2.41 M tokens (147× at 16 K) |
MoE backend FLASHINFER_CUTLASS, attention TRITON_MLA, NVFP4 linear
FlashInferCuteDslNvfp4W4A16, CUDA graphs + compile (4.2× vs eager), async scheduling.
Use
git clone https://github.com/r0b0tlab/ling30vl-nvfp4-mp-sm121.git
cd ling30vl-nvfp4-mp-sm121 && ./scripts/click-run.sh
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
model="ling-3.0-flash-vl-nvfp4-mp",
messages=[{"role": "user", "content": "What is 17*23? Think briefly."}],
temperature=0, max_tokens=512)
print(r.choices[0].message.content)
Thinking is on by default; pass chat_template_kwargs={"enable_thinking": False} for
deterministic short answers. Tool calling: --enable-auto-tool-choice --tool-call-parser ling3
(already set in the container).
Boundaries
- NVFP4 KV cache is unavailable for this dense-MLA model on SM121 (runtime-verified; fp8 is the maximum).
- Video input is not supported by the fork's VL implementation (image-only).
- n-gram speculative decoding is a throughput regression on general traffic (opt-in only).
- No MTP head ships in this checkpoint.
Citation / attribution
Quantized by r0b0tlab with NVIDIA Model Optimizer; base model by inclusionAI. Independent
community quantization — not affiliated with or endorsed by inclusionAI or NVIDIA. Evidence
bundle: evidence/ in the repro repository (sanitized; no prompts or responses).