This model applies Nota AI's non-uniform expert pruning (20% of routed experts removed) to nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.
Korean queries answered in Korean: 76.2% → 91.6% with a 58 GiB checkpoint (original: 70 GiB)
Average accuracy: English 84.0 → 83.6, Korean 71.4 → 72.9 (original NVFP4 → this model)
num_experts_per_layer in config.jsonvLLM needs a small patch so that MoE layers may have different numbers of experts.
uv venv --python 3.12 venv
source venv/bin/activate
uv pip install vllm==0.31.0
cd patch
python apply_patch.py --dry-run # show the changes
python apply_patch.py # apply the patch (originals are backed up to vllm/.nota_patch_backup/)
python apply_patch.py --revert # restore the original files
After applying the patch, serve the model on 1 x B200 with the arguments of the original model card (1 x GB200 recipe: MTP, reasoning parser, tool-call parser):
vllm serve nota-ai/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-Global-Pruned-20 \
--served-model-name nemotron-3.5-super-vl \
--tensor-parallel-size 1 \
--trust-remote-code \
--max-num-seqs 512 \
--enable-prefix-caching \
--max-num-batched-tokens 32768 \
--prefix-match-unit 16 \
--enable-mamba-shared-prefix-checkpoint \
--async-scheduling \
--kernel-config '{"linear_backend_per_quant":{"fp8_w8a8":"torch"}}' \
--compilation-config '{"pass_config":{"fuse_attn_quant":true},"use_inductor_graph_partition":true,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--mamba-backend flashinfer \
--mamba-ssm-cache-dtype float16 \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--mamba-cache-mode align \
--video-pruning-rate 0.5 \
--media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
--enable-auto-tool-choice \
--speculative_config.method mtp \
--speculative_config.num_speculative_tokens 5 \
--tool-call-parser qwen3_coder \
--reasoning-parser nemotron_v3
Reasoning is switched per request with chat_template_kwargs (e.g. {"enable_thinking": false}); see the
original model card for API client examples.
| Model | Pruning ratio | English avg. accuracy | Korean avg. accuracy | Korean language consistency |
|---|---|---|---|---|
| Original (NVFP4) | - | 84.0 | 71.4 | 76.2 |
| Uniform pruning | 20% | 82.3 | 68.2 | 75.9 |
| Ours | 20% | 83.6 | 72.9 | 91.6 |
| Configuration | Accuracy | Korean language consistency |
|---|---|---|
| Original (NVFP4) | 73.9 | 39.9 |
| Cluster-based non-uniform expert pruning | 73.7 | 41.7 |
| + Language-Following Expert Preservation | 82.3 | 51.7 |
| + Language-Following Weight Editing (this model) | 90.9 | 87.4 |
This model is a modified version of nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 and is distributed under the same license, the OpenMDW License Agreement, version 1.1. See the original model card for the base model's intended use, limitations, bias, safety and privacy notes.
model-00088-of-00088.safetensors 3.9 GBmodel-00001-of-00088.safetensors 2.9 GBmodel-00086-of-00088.safetensors 1.6 GBmodel-00002-of-00088.safetensors 1.6 GBmodel-00004-of-00088.safetensors 1.6 GBmodel-00018-of-00088.safetensors 1.3 GBmodel-00029-of-00088.safetensors 1.3 GBmodel-00033-of-00088.safetensors 1.3 GBmodel-00035-of-00088.safetensors 1.3 GBmodel-00040-of-00088.safetensors 1.3 GBmodel-00044-of-00088.safetensors 1.3 GBmodel-00051-of-00088.safetensors 1.3 GBmodel-00073-of-00088.safetensors 1.3 GBmodel-00031-of-00088.safetensors 1.3 GBmodel-00049-of-00088.safetensors 1.3 GBmodel-00080-of-00088.safetensors 1.3 GBmodel-00006-of-00088.safetensors 1.3 GBmodel-00013-of-00088.safetensors 1.3 GBmodel-00015-of-00088.safetensors 1.3 GBmodel-00038-of-00088.safetensors 1.3 GBmodel-00066-of-00088.safetensors 1.3 GBmodel-00068-of-00088.safetensors 1.3 GBmodel-00084-of-00088.safetensors 1.3 GBmodel-00020-of-00088.safetensors 1.3 GBmodel-00022-of-00088.safetensors 1.3 GBmodel-00024-of-00088.safetensors 1.3 GBmodel-00027-of-00088.safetensors 1.3 GBmodel-00042-of-00088.safetensors 1.3 GBmodel-00046-of-00088.safetensors 1.3 GBmodel-00055-of-00088.safetensors 1.3 GBmodel-00071-of-00088.safetensors 1.3 GBmodel-00011-of-00088.safetensors 1.2 GBmodel-00053-of-00088.safetensors 1.2 GBmodel-00057-of-00088.safetensors 1.2 GBmodel-00060-of-00088.safetensors 1.2 GBmodel-00062-of-00088.safetensors 1.2 GBmodel-00064-of-00088.safetensors 1.2 GBmodel-00075-of-00088.safetensors 1.2 GBmodel-00077-of-00088.safetensors 1.2 GBmodel-00082-of-00088.safetensors 1.2 GBmodel-00009-of-00088.safetensors 1.2 GBmodel-00012-of-00088.safetensors 110 MBmodel-00014-of-00088.safetensors 110 MBmodel-00016-of-00088.safetensors 110 MBmodel-00019-of-00088.safetensors 110 MBmodel-00021-of-00088.safetensors 110 MBmodel-00023-of-00088.safetensors 110 MBmodel-00025-of-00088.safetensors 110 MBmodel-00028-of-00088.safetensors 110 MBmodel-00030-of-00088.safetensors 110 MBmodel-00032-of-00088.safetensors 110 MBmodel-00034-of-00088.safetensors 110 MBmodel-00036-of-00088.safetensors 110 MBmodel-00039-of-00088.safetensors 110 MBmodel-00041-of-00088.safetensors 110 MBmodel-00043-of-00088.safetensors 110 MBmodel-00045-of-00088.safetensors 110 MBmodel-00047-of-00088.safetensors 110 MBmodel-00050-of-00088.safetensors 110 MBmodel-00052-of-00088.safetensors 110 MBmodel-00054-of-00088.safetensors 110 MBmodel-00056-of-00088.safetensors 110 MBmodel-00058-of-00088.safetensors 110 MBmodel-00061-of-00088.safetensors 110 MBmodel-00063-of-00088.safetensors 110 MBmodel-00065-of-00088.safetensors 110 MBmodel-00067-of-00088.safetensors 110 MBmodel-00069-of-00088.safetensors 110 MBmodel-00072-of-00088.safetensors 110 MBmodel-00074-of-00088.safetensors 110 MBmodel-00076-of-00088.safetensors 110 MBmodel-00078-of-00088.safetensors 110 MBmodel-00081-of-00088.safetensors 110 MBmodel-00083-of-00088.safetensors 110 MBmodel-00085-of-00088.safetensors 110 MBmodel-00087-of-00088.safetensors 110 MBmodel-00003-of-00088.safetensors 110 MBmodel-00005-of-00088.safetensors 110 MBmodel-00007-of-00088.safetensors 110 MBmodel-00010-of-00088.safetensors 110 MBmodel-00017-of-00088.safetensors 36 MBmodel-00026-of-00088.safetensors 36 MBmodel-00037-of-00088.safetensors 36 MBmodel-00048-of-00088.safetensors 36 MBmodel-00059-of-00088.safetensors 36 MBmodel-00070-of-00088.safetensors 36 MBmodel-00079-of-00088.safetensors 36 MBmodel-00008-of-00088.safetensors 36 MB