Rio-3.5-Open-397B-NVFP4

NVFP4-quantized build of prefeitura-rio/Rio-3.5-Open-397B — a 397B-parameter (17B active) Qwen3.5-MoE vision-language model (512 experts, hybrid softmax + linear/DeltaNet attention), itself a finetune of Qwen/Qwen3.5-397B-A17B. edit: ir daid ir was rhese thinfs. but it appears to be. nex n2 with a system prompt laugh out loud Quantized with NVIDIA TensorRT Model Optimizer 0.44 using the per-expert streaming calibration pipeline from local-inference-lab/quant-toolkit (Luke Alonso). Produced on 8x B200. Size on disk: ~251 GB (46 shards).

What's quantized

Only the routed MoE expert MLPs (gate/up/down) are NVFP4 (4-bit, blockwise FP8 scales, group size 16), calibrated per-expert. Left in BF16: shared-expert MLPs (active every token), attention (softmax + DeltaNet), router/gates, vision tower, MTP, embeddings, lm_head. KV cache is FP8 (e4m3). This mirrors lukealonso/Qwen3.5-397B-A17B-NVFP4.

• quant_algo: NVFP4 | quant_method: modelopt
• kv_cache_scheme: {'dynamic': False, 'num_bits': 8, 'type': 'float'}

Calibration

Per-expert max-calibration over a finetune-appropriate subset (Rio tracks the Qwen3.5 base, so the full base-model corpus is unnecessary): deep-reasoning + diverse-instruction + agentic-coding corpora (the same datasets as Luke's Qwen3.5 recipe). Post-calibration: rare-expert amaxes floored to median/10; gate/up weight amaxes tied for fused w13 export. Master amaxes published under calibration/ for reproducible re-export.

How to run (SGLang)

python3 -m sglang.launch_server \
  --model brandonmusic/Rio-3.5-Open-397B-NVFP4 \
  --served-model-name Rio-3.5 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --tensor-parallel-size 4 \
  --quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code \
  --speculative-algo NEXTN --speculative-num-steps 5 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \
  --mamba-scheduler-strategy extra_buffer \
  --mem-fraction-static 0.9 --host 0.0.0.0 --port 8000

Acknowledgements

• Luke Alonso — the per-expert NVFP4 quant-toolkit this build uses.
• prefeitura-rio — the base model, Rio-3.5-Open-397B.
• Qwen — Qwen3.5-397B-A17B, the underlying architecture.