Kimi-K3.1-Pro-Global-Pruned-37

This model applies Nota AI's non-uniform expert pruning (37.5% of routed experts removed) to moonshotai/Kimi-K3.1-Pro.

1M context on 8 x B200 (TP8) with a 948 GiB checkpoint, FP8 KV cache and DSpark speculative decoding
_{The original MXFP4 checkpoint (1,454 GiB) does not fit in 8 x B200 memory; the vLLM recipe serves it on 8 x B300. 1M uses --gpu-memory-utilization 0.985; all other arguments follow the vLLM recipe.}

• Routed experts: 82,432 → 51,520 (536–616 per layer; the first two and last two MoE layers keep all 896; layer 0 is dense)
• Experts per layer: text_config.num_experts_per_layer in config.json; kept expert ids: pruning_info.json
• Checkpoint size: 1,454 GiB → 948 GiB
• With the vLLM recipe settings plus FP8 KV cache (max-model-len 1,048,576), this model serves on NVIDIA B200 × 8 (TP8) with a KV cache of 1,415,974 tokens, speculative decoding and the vision encoder enabled, while the original checkpoint does not fit on B200 × 8
• Not supported with this checkpoint: expert parallel load balancing (EPLB) and the latency_specialized decode path

Installation

Kimi-K3.1-Pro support is not in a vLLM release yet; use the vllm/vllm-openai:kimi-k31 image (vLLM 0.1.dev24216+g266e88c05) and patch the vLLM package inside it.

cd patch
python apply_patch.py --dry-run     # show the changes
python apply_patch.py               # apply the patch (originals are backed up to vllm/.nota_patch_backup/)
python apply_patch.py --revert      # restore the original files

export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve <this repo> \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3 \
  --speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"method":"dspark","draft_sample_method":"probabilistic"}' \
  --kv-cache-dtype fp8 \
  --max-model-len 1048576 \
  --gpu-memory-utilization 0.985 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 64 \
  --disable-flashinfer-autotune

Performance (vs. the original)

• Max generation length: 65,536 tokens (including reasoning); reasoning effort max, temperature 1.0, top-p 0.95 (the settings of the Kimi-K3.1-Pro model card)
• Original: score reported in the Kimi-K3.1-Pro model card; the other benchmarks are not reported there, and the original does not fit on B200 × 8 for our own evaluation

Notification Commitment

The full Notification Commitment you agree to when requesting access to this repository is in NOTIFICATION_COMMITMENT.md. If any of its criteria apply to you, notify Nota at contact@nota.ai in advance. This commitment does not restrict, modify, or supersede the license below.

License

This model is licensed under the Kimi K3.1-Pro License (LICENSE).

You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.