NVFP4 quantization of Rakuten/RakutenAI-3.0 (671B MoE, DeepSeek-V3 architecture, MLA attention, 256 routed experts, MTP) with Four Over Six adaptive block scaling (arXiv:2512.02010).
lm_head and the MTP layer are kept in high precision.--quantization modelopt_fp4) and TensorRT-LLM on Blackwell GPUs without any code change.Four Over Six only changes how the per-block weight scale is chosen. For every 16-element weight block the scale target is picked between amax/6 and amax/4 by minimum squared error; 44.8% of blocks selected the amax/4 target. This lowers weight quantization MSE by 16.6% compared to plain ModelOpt NVFP4 round-to-nearest, while the packed bytes, scale tensors and metadata stay a standard NVFP4 checkpoint — no custom kernel, loader or runtime patch is required.
Two serving paths, both verified:
Blackwell (SM100+) — native NVFP4 (W4A4). Verified on 8×B300 with tensor parallel 8:
python3 -m sglang.launch_server \
--model-path rakuten-junliu/RakutenAI-3.0-NVFP4 \
--quantization modelopt_fp4 \
--tp 8 \
--trust-remote-code \
--mem-fraction-static 0.85 \
--port 30000
Hopper/Ampere (SM80–SM90) — Marlin kernels (W4A16). Weights stay 4-bit packed; compute runs in BF16, so this is not native NVFP4 execution, but the same checkpoint loads unmodified. Requires a recent sglang build with the Marlin FP4 backends. Verified on 8×H100-80GB:
python3 -m sglang.launch_server \
--model-path rakuten-junliu/RakutenAI-3.0-NVFP4 \
--quantization modelopt_fp4 \
--fp4-gemm-backend marlin \
--moe-runner-backend marlin \
--tp 8 \
--trust-remote-code \
--mem-fraction-static 0.85 \
--port 30000
curl -s http://127.0.0.1:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "default",
"messages": [{"role": "user", "content": "あなたは誰ですか?"}],
"max_tokens": 256,
"temperature": 0
}'
With the OpenAI Python client:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Explain mixture-of-experts routing in two sentences."}],
max_tokens=256,
temperature=0,
)
print(resp.choices[0].message.content)
| Item | Value |
|---|---|
| Native NVFP4 (W4A4) | Blackwell, SM100+ (B200 / B300) |
| Marlin fallback (W4A16) | SM80–SM90 (A100 / H100 / H200), verified on 8×H100-80GB |
| Checkpoint size | ~368 GB |
| Tested topology | 8 GPUs, --tp 8 |
Leave headroom for the KV cache: --mem-fraction-static 0.85 was used in all evaluation runs.
Measured with sglang TP8, greedy decoding.
| Model | GSM8K (5-shot) | MMLU |
|---|---|---|
| FP8 baseline | 94.39 | 85.27 |
| RakutenAI-3.0-NVFP4 (Four Over Six) | 95.00 | 84.69 |
License: Apache 2.0, same as the base model.
model-00001-of-000163.safetensors 4.7 GBmodel-00160-of-000163.safetensors 3.6 GBmodel-00006-of-000163.safetensors 2.7 GBmodel-00031-of-000163.safetensors 2.7 GBmodel-00053-of-000163.safetensors 2.7 GBmodel-00075-of-000163.safetensors 2.7 GBmodel-00097-of-000163.safetensors 2.7 GBmodel-00119-of-000163.safetensors 2.7 GBmodel-00023-of-000163.safetensors 2.7 GBmodel-00045-of-000163.safetensors 2.7 GBmodel-00067-of-000163.safetensors 2.7 GBmodel-00089-of-000163.safetensors 2.7 GBmodel-00111-of-000163.safetensors 2.7 GBmodel-00133-of-000163.safetensors 2.7 GBmodel-00152-of-000163.safetensors 2.7 GBmodel-00037-of-000163.safetensors 2.7 GBmodel-00059-of-000163.safetensors 2.7 GBmodel-00081-of-000163.safetensors 2.7 GBmodel-00103-of-000163.safetensors 2.7 GBmodel-00125-of-000163.safetensors 2.7 GBmodel-00144-of-000163.safetensors 2.7 GBmodel-00026-of-000163.safetensors 2.7 GBmodel-00048-of-000163.safetensors 2.7 GBmodel-00070-of-000163.safetensors 2.7 GBmodel-00092-of-000163.safetensors 2.7 GBmodel-00114-of-000163.safetensors 2.7 GBmodel-00136-of-000163.safetensors 2.7 GBmodel-00155-of-000163.safetensors 2.7 GBmodel-00040-of-000163.safetensors 2.7 GBmodel-00062-of-000163.safetensors 2.7 GBmodel-00084-of-000163.safetensors 2.7 GBmodel-00106-of-000163.safetensors 2.7 GBmodel-00128-of-000163.safetensors 2.7 GBmodel-00147-of-000163.safetensors 2.7 GBmodel-00021-of-000163.safetensors 2.7 GBmodel-00029-of-000163.safetensors 2.7 GBmodel-00051-of-000163.safetensors 2.7 GBmodel-00073-of-000163.safetensors 2.7 GBmodel-00095-of-000163.safetensors 2.7 GBmodel-00117-of-000163.safetensors 2.7 GBmodel-00139-of-000163.safetensors 2.7 GBmodel-00158-of-000163.safetensors 2.7 GBmodel-00043-of-000163.safetensors 2.7 GBmodel-00065-of-000163.safetensors 2.7 GBmodel-00087-of-000163.safetensors 2.7 GBmodel-00109-of-000163.safetensors 2.7 GBmodel-00131-of-000163.safetensors 2.7 GBmodel-00150-of-000163.safetensors 2.7 GBmodel-00015-of-000163.safetensors 2.7 GBmodel-00009-of-000163.safetensors 2.7 GBmodel-00018-of-000163.safetensors 2.7 GBmodel-00004-of-000163.safetensors 2.7 GBmodel-00035-of-000163.safetensors 2.7 GBmodel-00057-of-000163.safetensors 2.7 GBmodel-00079-of-000163.safetensors 2.7 GBmodel-00101-of-000163.safetensors 2.7 GBmodel-00123-of-000163.safetensors 2.7 GBmodel-00142-of-000163.safetensors 2.7 GBmodel-00013-of-000163.safetensors 2.7 GBmodel-00007-of-000163.safetensors 2.4 GBmodel-00033-of-000163.safetensors 2.4 GBmodel-00042-of-000163.safetensors 2.4 GBmodel-00055-of-000163.safetensors 2.4 GBmodel-00064-of-000163.safetensors 2.4 GBmodel-00077-of-000163.safetensors 2.4 GBmodel-00086-of-000163.safetensors 2.4 GBmodel-00099-of-000163.safetensors 2.4 GBmodel-00108-of-000163.safetensors 2.4 GBmodel-00121-of-000163.safetensors 2.4 GBmodel-00130-of-000163.safetensors 2.4 GBmodel-00149-of-000163.safetensors 2.4 GBmodel-00025-of-000163.safetensors 2.4 GBmodel-00028-of-000163.safetensors 2.4 GBmodel-00039-of-000163.safetensors 2.4 GBmodel-00047-of-000163.safetensors 2.4 GBmodel-00050-of-000163.safetensors 2.4 GBmodel-00061-of-000163.safetensors 2.4 GBmodel-00069-of-000163.safetensors 2.4 GBmodel-00072-of-000163.safetensors 2.4 GBmodel-00083-of-000163.safetensors 2.4 GBmodel-00091-of-000163.safetensors 2.4 GBmodel-00094-of-000163.safetensors 2.4 GBmodel-00105-of-000163.safetensors 2.4 GBmodel-00113-of-000163.safetensors 2.4 GBmodel-00116-of-000163.safetensors 2.4 GBmodel-00127-of-000163.safetensors 2.4 GBmodel-00135-of-000163.safetensors 2.4 GBmodel-00138-of-000163.safetensors 2.4 GBmodel-00146-of-000163.safetensors 2.4 GBmodel-00154-of-000163.safetensors 2.4 GBmodel-00157-of-000163.safetensors 2.4 GBmodel-00036-of-000163.safetensors 2.4 GBmodel-00058-of-000163.safetensors 2.4 GBmodel-00080-of-000163.safetensors 2.4 GBmodel-00102-of-000163.safetensors 2.4 GBmodel-00124-of-000163.safetensors 2.4 GBmodel-00143-of-000163.safetensors 2.4 GBmodel-00022-of-000163.safetensors 2.4 GBmodel-00044-of-000163.safetensors 2.4 GBmodel-00066-of-000163.safetensors 2.4 GBmodel-00088-of-000163.safetensors 2.4 GBmodel-00110-of-000163.safetensors 2.4 GBmodel-00132-of-000163.safetensors 2.4 GBmodel-00151-of-000163.safetensors 2.4 GBmodel-00030-of-000163.safetensors 2.4 GBmodel-00052-of-000163.safetensors 2.4 GBmodel-00074-of-000163.safetensors 2.4 GBmodel-00096-of-000163.safetensors 2.4 GBmodel-00118-of-000163.safetensors 2.4 GBmodel-00140-of-000163.safetensors 2.4 GBmodel-00159-of-000163.safetensors 2.4 GBmodel-00041-of-000163.safetensors 2.4 GBmodel-00063-of-000163.safetensors 2.4 GBmodel-00085-of-000163.safetensors 2.4 GBmodel-00107-of-000163.safetensors 2.4 GBmodel-00129-of-000163.safetensors 2.4 GBmodel-00148-of-000163.safetensors 2.4 GBmodel-00027-of-000163.safetensors 2.4 GBmodel-00049-of-000163.safetensors 2.4 GBmodel-00071-of-000163.safetensors 2.4 GBmodel-00093-of-000163.safetensors 2.4 GBmodel-00115-of-000163.safetensors 2.4 GBmodel-00137-of-000163.safetensors 2.4 GBmodel-00156-of-000163.safetensors 2.4 GBmodel-00038-of-000163.safetensors 2.4 GBmodel-00060-of-000163.safetensors 2.4 GBmodel-00082-of-000163.safetensors 2.4 GBmodel-00104-of-000163.safetensors 2.4 GBmodel-00126-of-000163.safetensors 2.4 GBmodel-00145-of-000163.safetensors 2.4 GBmodel-00024-of-000163.safetensors 2.4 GBmodel-00046-of-000163.safetensors 2.4 GBmodel-00068-of-000163.safetensors 2.4 GBmodel-00090-of-000163.safetensors 2.4 GBmodel-00112-of-000163.safetensors 2.4 GBmodel-00134-of-000163.safetensors 2.4 GBmodel-00153-of-000163.safetensors 2.4 GBmodel-00032-of-000163.safetensors 2.4 GBmodel-00054-of-000163.safetensors 2.4 GBmodel-00076-of-000163.safetensors 2.4 GBmodel-00098-of-000163.safetensors 2.4 GBmodel-00120-of-000163.safetensors 2.4 GBmodel-00020-of-000163.safetensors 2.4 GBmodel-00003-of-000163.safetensors 2.4 GBmodel-00011-of-000163.safetensors 2.4 GBmodel-00017-of-000163.safetensors 2.4 GBmodel-00008-of-000163.safetensors 2.4 GBmodel-00014-of-000163.safetensors 2.4 GBmodel-00005-of-000163.safetensors 2.4 GBmodel-00019-of-000163.safetensors 2.4 GBmodel-00002-of-000163.safetensors 2.4 GBmodel-00010-of-000163.safetensors 2.4 GBmodel-00016-of-000163.safetensors 2.4 GBmodel-00141-of-000163.safetensors 1.8 GBmodel-00034-of-000163.safetensors 983 MBmodel-00056-of-000163.safetensors 983 MBmodel-00078-of-000163.safetensors 983 MBmodel-00100-of-000163.safetensors 983 MBmodel-00122-of-000163.safetensors 983 MBmodel-00012-of-000163.safetensors 743 MBmodel-00161-of-000163.safetensors 0.1 kBmodel-00162-of-000163.safetensors 0.1 kBmodel-00163-of-000163.safetensors 0.1 kB有文件?提交种子。
提交种子Rakuten/RakutenAI-3.0