lukealonso/MiniMax-M2.7-NVFP4

Update 4/15/26 - Calibration data updated, KLD reduced by ~10%.

Update 4/12/26 - Calibration data updated, KLD reduced by ~20%.

Note: If you're experiencing issues with spurious spaces after punctuation, try downgrading transformers to 0.4.67

Model Description

MiniMax-M2.7-NVFP4 is an NVFP4-quantized version of MiniMaxAI/MiniMax-M2.7, a 230B-parameter Mixture-of-Experts language model with 10B active parameters.

The original model weights were converted from the official FP8 checkpoint to BF16, then quantized to NVFP4 (4-bit with blockwise FP8 scales per 16 elements) using NVIDIA Model Optimizer.

What's quantized

Only the MoE expert MLP layers (gate, up, and down projections) are quantized to NVFP4. All other layers are left in BF16. Since the expert weights constitute the vast majority of model parameters in an MoE architecture, this still yields significant memory savings.

Calibration uses natural top-k routing rather than forcing all experts to activate, so each expert's quantization scales reflect the token distributions it actually sees during inference. To compensate, calibration was run on a vastly larger number of samples than typical to ensure broad expert coverage through natural routing alone.

Calibration dataset

Samples were drawn from a diverse mix of publicly available datasets spanning code generation, function/tool calling, multi-turn reasoning, math, and multilingual (English + Chinese) instruction following. System prompts were randomly varied across samples. The dataset was designed to broadly exercise the model's capabilities and activate diverse token distributions across expert modules.

Running

exec docker run \
    --name sglang-m27a \
    --ipc=host \
    --shm-size=12g \
    --network=host \
    --cpuset-cpus=0-31 \
    --ulimit memlock=-1 \
    --ulimit stack=67108864 \
    --ulimit nofile=1048576:1048576 \
    --restart unless-stopped \
    -e SGLANG_USE_MESSAGE_QUEUE_BROADCASTER=1 \
    -e SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 \
    -e SGLANG_CUSTOM_ALLREDUCE_ALGO=oneshot \
    -e SGLANG_DISABLE_FA4_WARMUP=1 \
    -e SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD=4096 \
    -e SGLANG_ENABLE_JIT_DEEPGEMM=0 \
    -e NCCL_IB_DISABLE=1 \
    -e NCCL_P2P_DISABLE=1 \
    -e NCCL_NVLS_ENABLE=0 \
    -e NCCL_CUMEM_ENABLE=0 \
    -e NCCL_P2P_LEVEL=SYS \
    -e B12X_MOE_FORCE_A16=1 \
    -e NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 \
    -e NCCL_MIN_NCHANNELS=8 \
    -e NCCL_SOCKET_NTHREADS=4 \
    -e NCCL_NSOCKS_PERTHREAD=2 \
    -e NCCL_BUFFSIZE=16777216 \
    -e TORCH_NCCL_AVOID_RECORD_STREAMS=1 \
    -e OMP_NUM_THREADS=16 \
    -e MKL_NUM_THREADS=16 \
    -e OPENBLAS_NUM_THREADS=16 \
    -e NUMEXPR_NUM_THREADS=16 \
    -e TOKENIZERS_PARALLELISM=false \
    -e CUDA_DEVICE_MAX_CONNECTIONS=1 \
    -e CUDA_MODULE_LOADING=LAZY \
    -e SAFETENSORS_FAST_GPU=1 \
    -e TRITON_CACHE_DIR=/cache/triton \
    -e TORCH_COMPILE_DEBUG=0 \
    -e HF_HUB_ENABLE_HF_TRANSFER=1 \
    -e HF_HOME=/root/.cache/huggingface \
    -e TRANSFORMERS_OFFLINE=1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
       llm-sglang-blackwell:cu130 \
    python -m sglang.launch_server \
    --grammar-backend none \
    --model lukealonso/MiniMax-M2.7-NVFP4 \
    --served-model-name MiniMax-M2.7 \
    --tensor-parallel-size 2 \
    --quantization modelopt_fp4 \
    --kv-cache-dtype bfloat16 \
    --dtype auto \
    --prefill-max-requests 4 \
    --stream-interval 16 \
    --load-format safetensors \
    --trust-remote-code \
    --context-length 196608 \
    --mem-fraction-static 0.93 \
    --chunked-prefill-size 4096 \
    --max-prefill-tokens 4096 \
    --disable-radix-cache \
    --schedule-conservativeness 0.40 \
    --max-running-requests 8 \
    --cuda-graph-max-bs 8 \
    --sampling-backend flashinfer \
    --cuda-graph-bs 1 2 4 6 8 \
    --num-continuous-decode-steps 4 \
    --enable-mixed-chunk \
    --attention-backend flashinfer \
    --moe-runner-backend b12x \
    --fp4-gemm-backend b12x \
    --enable-pcie-oneshot-allreduce \
    --pcie-oneshot-allreduce-max-size 8388608 \
    --tool-call-parser minimax-m2 \
    --reasoning-parser minimax-append-think \
    --host 127.0.0.1 \
    --hicache-size 36 \
    --hicache-io-backend kernel \
    --hicache-mem-layout page_first \
    --hicache-write-policy write_through_selectitive

Tested on 2x and 4x RTX Pro 6000 Blackwell.

  Docker Image: voipmonitor/sglang:cu130 (festr, 6 days old, has b12x built-in)             
  Model: lukealonso/GLM-5.1-NVFP4 (434 GB, glm_moe_dsa, 78 layers, 256 experts)   
          
  Launch command:                 
  export OMP_NUM_THREADS=16       
  export SGLANG_ENABLE_SPEC_V2=True                       
  export NVIDIA_VISIBLE_DEVICES=1,2,3,4,5,6,7,8  # 8x Blackwell                   
          
  python -m sglang.launch_server \
    --model-path lukealonso/MiniMax-M2.7-NVFP4 \        
    --served-model-name MiniMax-M2.7 \   
    --reasoning-parser minimax \
    --tool-call-parser minimax-m2 \   
    --tp 2 \
    --enable-torch-compile \
    --trust-remote-code \           
    --quantization modelopt_fp4 \   
    --kv-cache-dtype bf16 \
    --moe-runner-backend b12x \
    --fp4-gemm-backend b12x \       
    --attention-backend flashinfer \
    --enable-pcie-oneshot-allreduce \                    
    --mem-fraction-static 0.85 \                          
    --host 0.0.0.0 --port 5000