llm-jp-4-32b-a3b-thinking-NVFP4

NVFP4 (W4A4) quantization of llm-jp/llm-jp-4-32b-a3b-thinking — a Japanese-native 32B-A3B Mixture-of-Experts thinking (reasoning) model. Quantized with llm-compressor 0.12.0 to the compressed-tensors nvfp4-pack-quantized format, ready to serve on NVIDIA Blackwell (SM120) GPUs via vLLM with the fast FLASHINFER_CUTLASS MoE kernel + CUDA graphs.

• ~20.7 GB (vs ~64 GB bf16). Experts NVFP4 4-bit; lm_head kept in bf16; router/gates excluded.
• Reasoning survives quantization and Japanese stays fluent; in internal testing it also writes look-ahead-safe trading/backtest code (uses .shift(1) for next-bar execution).

⚠️ Important: padded MoE intermediate (960 → 1024)

The base model's moe_intermediate_size is 960, which is not accepted by vLLM's fast NVFP4 MoE kernels (FLASHINFER_CUTLASS/MARLIN require 128-aligned, and 960/TP=480 is rejected).

This checkpoint therefore has each expert's intermediate dimension zero-padded from 960 to 1024 before quantization, and config.json reports moe_intermediate_size: 1024. The padding is mathematically lossless (the 64 extra units have zero gate/up output rows and zero down-projection columns, contributing exactly 0), but the tensor shapes differ from the base model — keep this in mind if you diff against llm-jp/llm-jp-4-32b-a3b-thinking.

Serving (vLLM ≥ 0.22, Blackwell)

vllm serve sakamakismile/llm-jp-4-32b-a3b-thinking-NVFP4 \
  --trust-remote-code \
  --tensor-parallel-size 2 \           # or 4
  --max-model-len 16384 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90
# multi-GPU without NVLink/P2P: add  --disable-custom-all-reduce  and env NCCL_P2P_DISABLE=1

• compressed-tensors quantization is auto-detected; FLASHINFER_CUTLASS NVFP4 MoE backend is selected automatically on Blackwell.
• Reasoning format: the model uses the harmony channel format (analysis → final). vLLM 0.22's openai_gptoss reasoning parser raises NotImplementedError in non-streaming mode, so run without --reasoning-parser — the harmony text appears in content and the user-facing answer follows the final channel marker. (Use streaming if you want the parser to split reasoning_content.)
• A custom tokenizer (llmjp4_tokenizer.py / llmjp4_harmony.py) ships with the model and loads via --trust-remote-code.

Measured throughput

On 2× / 4× NVIDIA RTX PRO 2000 Blackwell (16 GB, 288 GB/s, no NVLink), kv fp8, CUDA graphs:

| | single-stream | aggregate |

|---|---|---|

| TP=2 | ~119 tok/s | ~295 tok/s (4-way) |

| TP=4 | ~151–185 tok/s | ~465 (4-way) / ~750 (8-way) tok/s |

Quantization recipe

QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head", "re:.mlp.gate$", "re:.mlp.shared_expert_gate$"]), calibrated on neuralmagic/calibration (LLM split). MoE experts are linearized for per-expert calibration (calibrate_all_experts).

License & attribution

Apache-2.0, inherited from the base model llm-jp/llm-jp-4-32b-a3b-thinking by LLM-jp. Please cite and follow the base model's terms. This is a community quantization; no affiliation with LLM-jp.