RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8

Model Overview

• Model Architecture: NemotronH_Omni_Reasoning_V3
  • Input: Text/Image
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP8
  • Activation quantization: FP8
• Configured Maximum Context Length: 1,048,576 tokens
• Release Date: 2026-10-15
• Version: 1.0
• Model Developers: RedHatAI

This is an FP8 Dynamic variant of the nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 checkpoint, prepared for vLLM inference.

The following sections are covered below:

• Model Optimizations
• Evaluation
  • Accuracy metrics
  • Performance Benchmarks
  • Combining with a DSpark speculator for further acceleration
• Deployment
  • vLLM Serving
• Creation script using LLM Compressor

Model Optimizations

LLM Compressor model-free PTQ applies FP8 Dynamic quantization to selected linear operators in the backbone and MTP modules. Quantized weights use per-channel FP8, and activations use dynamic per-token FP8 scaling. Other tensors retain their original precision. The checkpoint is saved in compressed-tensors format.

Quantized weights use 8 bits instead of 16-bit BF16, reducing storage and memory for those tensors. Overall savings depend on the share of the model that remains in BF16.

Evaluation

Accuracy

To assess the impact of quantization on model accuracy, particularly in reasoning-intensive tasks and agentic workflows, we evaluate both the BF16 and FP8 models across a broad suite of representative benchmarks. To ensure a fair and consistent comparison, both models are evaluated under identical conditions: non-greedy sampling, 4× NVIDIA H200 GPUs, vLLM 0.31.0, and the Inspect AI and Harbor evaluation frameworks. The evaluation results are presented in the figure below.

The FP8 checkpoint consistently preserves model accuracy across all evaluated benchmarks.

Performance Benchmarks

The plots demonstrate the performance boost given to the Nemotron 3.5 Super VL (4x H100, temperature 0, max_tokens 1024) when swapped from the original BF16 checkpoint to the FP8 quantized counterpart (this model). Both models were served with MTP using 3 draft tokens, across workloads with prompts from various datasets. The FP8 checkpoint delivers a ~1.2x speedup for a single user, rising to ~1.3-1.4x under concurrent load. As the number of speculative tokens is unchanged, the gain comes purely from faster quantized compute. Each chart plots interactivity, defined as 1000 divided by mean inter-token latency (i.e., the tok/s streaming speed a single user experiences), on the x-axis against aggregate output throughput (tok/s) on the y-axis, with one point per closed-loop concurrency level N. The HumanEval prompts come from OpenAI's HumanEval coding benchmark, while the math_reasoning and RAG prompts come from Spec-Bench, the ACL 2024 speculative decoding benchmark suite.

Combining with a DSpark speculator

For further performance gains, the DSpark speculator RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.dspark can be used in place of MTP. The plots below replicate the above benchmarking setup, but swap to use the DSpark speculator with 8 speculative tokens. See the speculator's model card for serving details.

Deployment

vLLM Serving

The following example uses MTP speculative decoding and FlashInfer for the Mamba backend:

vllm serve RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.89 \
  --prefix-match-unit 64 \
  --block-size 64 \
  --enable-mamba-fine-grained-prefix-cache \
  --compilation_config.max_cudagraph_capture_size 192 \
  --enable-expert-parallel \
  --max-num-seqs 128 \
  --trust-remote-code \
  --max-model-len 114688 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --async-scheduling \
  --speculative_config.method mtp \
  --speculative_config.num_speculative_tokens 3 \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --chat-template-content-format string \
  --mamba-cache-mode align \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

The FP8 checkpoint is documented to fit on 2x H100 GPUs, compared with 4 GPUs for the BF16 checkpoint. Hardware needs vary with serving configuration and workload.

Creation

This checkpoint was produced with LLM Compressor's model-free PTQ flow using the FP8_DYNAMIC scheme. No calibration dataset is required. The script below uses a BF16 checkpoint as input and saves the quantized checkpoint in compressed-tensors format.

Quantization script

"""Apply FP8 Dynamic to the Nemotron model and selected MTP modules."""

import json
from pathlib import Path

from compressed_tensors.quantization import QuantizationConfig
from llmcompressor import model_free_ptq


# Set these paths before running the example.
MODEL_ID = "/path/to/bf16-checkpoint"
SAVE_DIR = "/path/to/fp8-dynamic-checkpoint"
DEVICE = "cuda:0"

# Match non-MTP modules, then let IGNORE restrict the backbone to eligible linears.
TARGETS = [
    # Backbone modules, excluding MTP modules.
    r"re:^(?!.*(?:^|\.)mtp\.).+$",
    # MTP routed-expert gate, up, and down projections.
    r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.experts\.\d+\."
    r"(gate_proj|up_proj|down_proj)$",
    # MTP attention q, k, v, and output projections.
    r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\."
    r"(q_proj|k_proj|v_proj|o_proj)$",
    # MTP shared-expert gate, up, and down projections.
    r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.shared_experts\."
    r"(gate_proj|up_proj|down_proj)$",
]

IGNORE = [
    # Standard exclusions.
    r"re:.*lm_head$",
    r"re:.*conv1d.*",
    r"re:.*backbone\.embeddings$",
    r"re:.*_latent_proj$",
    r"re:.*mixer\.gate$",
    # Vision model and connector.
    r"re:^vision_model\..*",
    r"re:^mlp1\..*",
    # Norms.
    r"re:.*backbone\.norm_f$",
    r"re:.*\.norm[12]$",
    # Backbone shared experts; MTP shared experts remain targeted.
    r"re:.*(?:^|\.)(?:backbone|model)\.layers\.\d+\.mixer\."
    r"shared_experts(?:\.|$).*",
    # MTP routers and prediction head.
    r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.gate$",
    r"re:.*(?:^|\.)mtp\.layers\.\d+\.eh_proj$",
]

config = QuantizationConfig(
    config_groups={"FP8_DYNAMIC": TARGETS},
    ignore=IGNORE,
)

model_free_ptq(
    model_stub=MODEL_ID,
    save_directory=SAVE_DIR,
    config=config,
    device=DEVICE,
)

# Keep the top-level and nested quantization metadata in sync for MTP.
config_path = Path(SAVE_DIR) / "config.json"
saved_config = json.loads(config_path.read_text())
if "llm_config" in saved_config:
    saved_config["llm_config"]["quantization_config"] = saved_config[
        "quantization_config"
    ]
    config_path.write_text(json.dumps(saved_config, indent=2) + "\n")