AxionML Nemotron-3.5-Lightning-30B-A3B-NVFP4

Mirrored by AxionML for open-source serving and deployment use cases. Part of AxionML's effort to provide ready-to-serve quantized models for the community.

Released by NVIDIA. The weights in this repository are an unmodified copy of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (revision bee7596271d1495f6992ae224aefde4410e816b8). All credit belongs to NVIDIA.

NVIDIA Nemotron 3.5 Lightning is a hybrid Mamba-2 / MoE / attention model (30B total, 3B active) built for long-running agents and efficient local inference. NVIDIA pre-trained it with an NVFP4 recipe and publishes this NVFP4 checkpoint directly.

About NVFP4 quantization: NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity while higher-precision FP32 accumulation protects dot-product accuracy.

Ready for commercial and non-commercial use under OpenMDW-1.1.

Model Summary

Evaluation Results

Scores reported by NVIDIA (NeMo Gym / NeMo Evaluator). Recommended sampling: temperature=1.0, top_p=0.95.

Quantization Details

• Quantization format: Four-Over-Six NVFP4 (static MSE-style calibration) W4A16 on routed and shared experts; FP8 per-tensor dynamic scales on Mamba in_proj / out_proj
• KV cache: FP8
• Calibration: 1,000 samples at 32K tokens from a Nemotron Ultra validation subset
• Tool: NVIDIA Model Optimizer

Usage

Deploy with SGLang

sglang serve \
    --model-path AxionML/Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
    --mamba-backend flashinfer \
    --mamba-ssm-dtype float16 \
    --enable-mamba-cache-stochastic-rounding \
    --mamba-cache-philox-rounds 5 \
    --mem-fraction-static 0.85 \
    --cuda-graph-max-bs-decode 16 \
    --reasoning-parser nemotron_3 \
    --tool-call-parser qwen3_coder

Deploy with vLLM

vllm serve AxionML/Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.85 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark \
    --speculative_config.num_speculative_tokens 3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

SGLang command is the upstream 1x B200 recipe (lmsysorg/sglang:dev-nemotron3-5-lightning); the vLLM command is the upstream DGX Spark recipe with DSpark (vllm/vllm-openai:v0.27.1). For MTP in SGLang add --speculative-algorithm EAGLE --speculative-draft-model-path AxionML/Nemotron-3.5-Lightning-30B-A3B-NVFP4 --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6.

Limitations

The base model was trained on data that may contain toxic language and societal biases. The quantized model inherits these limitations. It may generate inaccurate, biased, or offensive content. Please refer to the original model card and the upstream quantized model card for full details.

Credits

• Base model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
• Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 by NVIDIA
• Mirror: AxionML