中文Sign up

sakamakismile/llm-jp-4-32b-a3b-thinking-NVFP4

🤗 Hugging Face sourcetext-generationapache-2.019.7B params3B activated21 GBsafetensors✓ 2 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo sakamakismile/llm-jp-4-32b-a3b-thinking-NVFP4 ./model-folder
Needs a seeder →

llm-jp-4-32b-a3b-thinking-NVFP4

NVFP4 (W4A4) quantization of llm-jp/llm-jp-4-32b-a3b-thinking — a Japanese-native 32B-A3B Mixture-of-Experts thinking (reasoning) model. Quantized with llm-compressor 0.12.0 to the compressed-tensors nvfp4-pack-quantized format, ready to serve on NVIDIA Blackwell (SM120) GPUs via vLLM with the fast FLASHINFER_CUTLASS MoE kernel + CUDA graphs.

  • ~20.7 GB (vs ~64 GB bf16). Experts NVFP4 4-bit; lm_head kept in bf16; router/gates excluded.
  • Reasoning survives quantization and Japanese stays fluent; in internal testing it also writes look-ahead-safe trading/backtest code (uses .shift(1) for next-bar execution).

⚠️ Important: padded MoE intermediate (960 → 1024)

The base model's moe_intermediate_size is 960, which is not accepted by vLLM's fast NVFP4 MoE kernels (FLASHINFER_CUTLASS/MARLIN require 128-aligned, and 960/TP=480 is rejected).

This checkpoint therefore has each expert's intermediate dimension zero-padded from 960 to 1024 before quantization, and config.json reports moe_intermediate_size: 1024. The padding is mathematically lossless (the 64 extra units have zero gate/up output rows and zero down-projection columns, contributing exactly 0), but the tensor shapes differ from the base model — keep this in mind if you diff against llm-jp/llm-jp-4-32b-a3b-thinking.

Serving (vLLM ≥ 0.22, Blackwell)

vllm serve sakamakismile/llm-jp-4-32b-a3b-thinking-NVFP4 \
  --trust-remote-code \
  --tensor-parallel-size 2 \           # or 4
  --max-model-len 16384 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90
# multi-GPU without NVLink/P2P: add  --disable-custom-all-reduce  and env NCCL_P2P_DISABLE=1
  • compressed-tensors quantization is auto-detected; FLASHINFER_CUTLASS NVFP4 MoE backend is selected automatically on Blackwell.
  • Reasoning format: the model uses the harmony channel format (analysis → final). vLLM 0.22's openai_gptoss reasoning parser raises NotImplementedError in non-streaming mode, so run without --reasoning-parser — the harmony text appears in content and the user-facing answer follows the final channel marker. (Use streaming if you want the parser to split reasoning_content.)
  • A custom tokenizer (llmjp4_tokenizer.py / llmjp4_harmony.py) ships with the model and loads via --trust-remote-code.

Measured throughput

On 2× / 4× NVIDIA RTX PRO 2000 Blackwell (16 GB, 288 GB/s, no NVLink), kv fp8, CUDA graphs:

single-stream aggregate
TP=2 ~119 tok/s ~295 tok/s (4-way)
TP=4 ~151–185 tok/s ~465 (4-way) / ~750 (8-way) tok/s

Quantization recipe

QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"]), calibrated on neuralmagic/calibration (LLM split). MoE experts are linearized for per-expert calibration (calibrate_all_experts).

License & attribution

Apache-2.0, inherited from the base model llm-jp/llm-jp-4-32b-a3b-thinking by LLM-jp. Please cite and follow the base model's terms. This is a community quantization; no affiliation with LLM-jp.