KAT-Coder-V2.5-Dev-NVFP4

NVFP4 (W4A4, compressed-tensors) quantization of

Kwaipilot/KAT-Coder-V2.5-Dev —

the 35B-A3B agentic-coding MoE (SWE-bench Verified 69.4% upstream).

70GB bf16 → 21.9GB. Runs on a single 24GB Blackwell card, or 2× 16GB.

Quantized by Lna-Lab (@Tono_Ken3).

Measured (12x RTX PRO 2000 Blackwell 16GB, 60W power cap each)

• TP=2 (2 GPUs): ~122 tok/s single-stream
• Sanity: lookahead-bias trading question answered with the correct shift(1)

fix; clean O(n) implementations with tests.

Serving

vllm serve  --tensor-parallel-size 2 --max-model-len 32768 \
  --gpu-memory-utilization 0.92
# NVFP4 is auto-detected; no --quantization flag needed.
# On no-P2P multi-GPU boxes add: NCCL_P2P_DISABLE=1, --disable-custom-all-reduce

Recipe notes

• Arch qwen3_5_moe quantizes cleanly with llm-compressor

(targets=Linear, scheme NVFP4) with these ignores:

lm_head, re:.conv1d. (DeltaNet conv), re:.*mlp.gate$ and

re:.shared_expert_gate$ (MoE routers), re:.mtp.*.

• The open-weight release ships without vision and without MTP tensors

(we checked; nothing to graft).

• Calibration: 32 samples x 8192 tokens (neuralmagic/calibration), single GPU,

~4 minutes total wall.

• Newer transformers removed GraniteMoeParallelExperts which llm-compressor

still imports — a dummy class injected before import satisfies it safely for

non-Granite models.

W4A4 is stable on this architecture (5th model family we've confirmed:

Nex-N2, Holo, Ornith, AgentWorld, KAT).