RakutenAI-3.0-W4A8
Mixed-precision quantization of Rakuten/RakutenAI-3.0 (671B MoE, DeepSeek-V3 architecture, MLA attention, 256 routed experts, MTP):
- Routed experts: INT4 weights, group size 128 (
W4A8_AWQ), FP8 activations with static per-tensor scales. - Attention, shared experts and the first 3 dense layers: original FP8 block scales (
FP8_BLOCK_SCALES). - Checkpoint format: TensorRT-LLM
MIXED_PRECISION(w4afp8in sglang). - Calibration: a mixed Japanese/English calibration set including identity/safety prompts, instruction data, Wikipedia and news text, 820 samples.
Quantization method
Activation scales come from TensorRT-Model-Optimizer FP8 per-tensor calibration over the calibration set above; routed-expert weights are quantized to INT4 with group size 128 and exported through the TensorRT-LLM mixed-precision MoE export path. Everything outside the routed experts keeps the FP8 block-scale weights of the source model.
How to run (sglang)
Requires Hopper GPUs (SM90). Verified on 8×H100-80GB with tensor parallel 8.
python3 -m sglang.launch_server \
--model-path rakuten-junliu/RakutenAI-3.0-W4A8 \
--quantization w4afp8 \
--tp 8 \
--trust-remote-code \
--mem-fraction-static 0.85 \
--port 30000
Wait until curl -sf http://127.0.0.1:30000/health_generate succeeds before sending traffic (first startup loads ~341 GB of weights).
Query the server
curl -s http://127.0.0.1:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "default",
"messages": [{"role": "user", "content": "あなたは誰ですか?"}],
"max_tokens": 256,
"temperature": 0
}'
With the OpenAI Python client:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Explain mixture-of-experts routing in two sentences."}],
max_tokens=256,
temperature=0,
)
print(resp.choices[0].message.content)
Hardware requirements
| Item | Value |
|---|---|
| GPU architecture | Hopper, SM90 (H100 / H200) only |
| Checkpoint size | ~341 GB |
| Tested topology | 8 GPUs, --tp 8 |
This checkpoint does not run on Blackwell. The CUTLASS W4A8 grouped-GEMM kernel used by the w4afp8 path is compiled for sm90 only, and server startup aborts on Blackwell with an arch-conditional MMA error. Use RakutenAI-3.0-NVFP4 on Blackwell instead.
Evaluation
Measured with sglang TP8, greedy decoding.
| Model | GSM8K (5-shot) | MMLU |
|---|---|---|
| FP8 baseline | 94.39 | 85.27 |
| RakutenAI-3.0-W4A8 | 94.39 | 85.26 |
GSM8K and MMLU are close to saturation for this model. Hard-suite results, measured with lm-eval 0.4.12, the checkpoint chat template, greedy decoding, seed 1, and a pinned client tokenizer, served with SGLang TP8 on 8×H100:
| Benchmark | Score |
|---|---|
| MMLU-Pro (5-shot) | 73.94% (8,897/12,032) |
| BBH (3-shot CoT) | 79.63% (5,185/6,511) |
| AIME 2024 (0-shot) | 16/30 |
License: Apache 2.0, same as the base model.