中文Sign up

brandonmusic/Rio-3.5-Open-397B-NVFP4

🤗 Hugging Face sourcetext-generationmit251 GBsafetensors✓ 49 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo brandonmusic/Rio-3.5-Open-397B-NVFP4 ./model-folder
Needs a seeder →

Rio-3.5-Open-397B-NVFP4

NVFP4-quantized build of prefeitura-rio/Rio-3.5-Open-397B — a 397B-parameter (17B active) Qwen3.5-MoE vision-language model (512 experts, hybrid softmax + linear/DeltaNet attention), itself a finetune of Qwen/Qwen3.5-397B-A17B. edit: ir daid ir was rhese thinfs. but it appears to be. nex n2 with a system prompt laugh out loud Quantized with NVIDIA TensorRT Model Optimizer 0.44 using the per-expert streaming calibration pipeline from local-inference-lab/quant-toolkit (Luke Alonso). Produced on 8x B200. Size on disk: ~251 GB (46 shards).

What's quantized

Only the routed MoE expert MLPs (gate/up/down) are NVFP4 (4-bit, blockwise FP8 scales, group size 16), calibrated per-expert. Left in BF16: shared-expert MLPs (active every token), attention (softmax + DeltaNet), router/gates, vision tower, MTP, embeddings, lm_head. KV cache is FP8 (e4m3). This mirrors lukealonso/Qwen3.5-397B-A17B-NVFP4.

  • quant_algo: NVFP4 | quant_method: modelopt
  • kv_cache_scheme: {'dynamic': False, 'num_bits': 8, 'type': 'float'}

Calibration

Per-expert max-calibration over a finetune-appropriate subset (Rio tracks the Qwen3.5 base, so the full base-model corpus is unnecessary): deep-reasoning + diverse-instruction + agentic-coding corpora (the same datasets as Luke's Qwen3.5 recipe). Post-calibration: rare-expert amaxes floored to median/10; gate/up weight amaxes tied for fused w13 export. Master amaxes published under calibration/ for reproducible re-export.

How to run (SGLang)

python3 -m sglang.launch_server \
  --model brandonmusic/Rio-3.5-Open-397B-NVFP4 \
  --served-model-name Rio-3.5 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --tensor-parallel-size 4 \
  --quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code \
  --speculative-algo NEXTN --speculative-num-steps 5 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \
  --mamba-scheduler-strategy extra_buffer \
  --mem-fraction-static 0.9 --host 0.0.0.0 --port 8000

Acknowledgements

  • Luke Alonso — the per-expert NVFP4 quant-toolkit this build uses.
  • prefeitura-rio — the base model, Rio-3.5-Open-397B.
  • Qwen — Qwen3.5-397B-A17B, the underlying architecture.