RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.dspark

Model Overview

Model Architecture: DSparkDraftModel
- Target Model Architecture: NemotronH_Omni_Reasoning_V3
- Input: Text/Image
- Output: Text
Model Optimizations:
- Speculative Decoding Algorithm: DSpark
- Draft Model: 5-layer Qwen3-style backbone
- Maximum Draft Length: 8 tokens
- Auxiliary Target Layers: 17, 37, 59, 79, and 87
- Markov Head: Vanilla, rank 256
- Confidence Head: Enabled with Markov features
Configured Maximum Context Length: 1,048,576 tokens
Release Date: 2026-10-15
Version: 1.0
Model Developers: RedHatAI

This is a DSpark speculator model for NVIDIA's Nemotron 3.5 Super VL checkpoint. It was trained using the Speculators library (version 0.9.0a69).

DSpark extends DFlash with a Markov head for modeling intra-block token dependencies and a confidence head for predicting per-position acceptance. The 5-layer draft model consumes auxiliary hidden states from configured target layer IDs 17, 37, 59, 79, and 87 and proposes up to 8 tokens per decoding step.

The following sections are covered below:

Evaluation
Deployment
- vLLM Serving
Training Details

Evaluation

Acceptance Rates

Evaluation used vLLM 0.31.0 and the nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 verifier on the Spec-Bench benchmark.

Per-position acceptance rates:

Temperature 0

Dataset	Acceptance Length	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7
HumanEval	5.73	91.1%	80.9%	70.4%	61.0%	52.4%	45.2%	39.0%	33.5%
math_reasoning	6.68	94.9%	88.6%	81.0%	74.2%	67.5%	60.3%	54.3%	47.4%
qa	3.63	79.4%	59.1%	42.5%	30.4%	20.8%	14.2%	9.9%	6.7%
question	3.96	80.7%	61.7%	45.7%	34.3%	26.3%	19.9%	15.2%	12.0%
rag	4.72	86.3%	72.6%	59.4%	47.1%	37.0%	29.5%	22.9%	17.6%
summarization	4.38	84.4%	68.7%	54.6%	42.6%	32.2%	24.3%	18.4%	13.2%
tool_call	4.71	85.1%	69.8%	56.3%	45.3%	36.8%	30.3%	25.7%	21.8%
translation	4.58	84.9%	70.9%	57.1%	45.5%	35.5%	27.7%	21.0%	15.7%
writing	3.92	80.0%	61.0%	44.8%	33.8%	25.9%	19.6%	15.1%	11.9%

Temperature 1

Dataset	Acceptance Length	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7
HumanEval	5.32	87.8%	75.6%	64.6%	54.7%	46.6%	39.6%	34.0%	28.8%
math_reasoning	6.34	92.6%	84.3%	76.6%	69.3%	62.3%	56.1%	49.4%	43.5%
qa	3.43	76.5%	55.5%	39.0%	27.0%	18.4%	12.6%	8.4%	5.5%
question	3.77	78.2%	58.3%	42.6%	31.7%	23.9%	18.0%	13.9%	10.8%
rag	4.37	83.6%	67.3%	53.3%	41.3%	32.5%	25.0%	19.2%	14.6%
summarization	4.15	82.3%	65.8%	51.0%	38.7%	28.9%	21.0%	15.7%	11.1%
tool_call	4.05	80.2%	62.0%	47.4%	36.0%	27.5%	21.2%	16.9%	13.5%
translation	4.35	83.3%	67.8%	53.3%	41.7%	32.1%	24.5%	18.3%	13.5%
writing	3.77	78.2%	58.3%	42.6%	31.7%	23.9%	18.0%	13.9%	10.8%

Long-Context Acceptance

Evaluation of RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.dspark using 8 speculative tokens on MRCR 8-Needle.

Prompt length	Unique requests	Mean acceptance length
4K–8K	97	6.276
8K–16K	93	5.754
16K–32K	99	5.922
32K–65K	94	5.817
65K–131K	108	5.794
131K–262K	96	6.018
262K–524K	102	6.303
524K–1M	81	4.082
1M+¹	9	3.561
Overall	779	5.719

Maximum successful prompt length within the retained window: 1,034,643 tokens

Performance Evaluations against MTP

The plots compare speculative decoding on Nemotron 3.5 Super VL (BF16, 4x H100, temperature 0, max_tokens 1024), with MTP using 3 draft tokens and DSpark using 8, across workloads with prompts from various datasets. Each chart plots interactivity, defined as 1000 divided by mean inter-token latency (i.e., the tok/s streaming speed a single user experiences), on the x-axis against aggregate output throughput (tok/s) on the y-axis, with one point per closed-loop concurrency level N. Where the DSpark curve sits above the MTP curve shows the concurrency levels at which deeper speculation buys better per-user latency and/or total throughput; where the curves converge, the server is saturated and extra speculation no longer helps. The HumanEval prompts come from OpenAI's HumanEval coding benchmark, while the math_reasoning and RAG prompts come from Spec-Bench, the ACL 2024 speculative decoding benchmark suite.

Combining with a quantized verifier for greater acceleration

Further acceleration can be achieved by swapping the BF16 checkpoint to a quantized verifier. In the plots below, we repeat the experiments from above but use the FP8 quantized checkpoint, RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8, demonstrating even greater performance gains.

Adaptive Verification

The DSpark speculator is trained and shipped with a confidence head. With adaptive verification on, num_speculative_tokens=8 sets the maximum draft length, but each step only verifies as many draft tokens as the cost model's budget allows. Per step, the draft model proposes up to 8 tokens, and the trained confidence head estimates each one's chance of acceptance. A token only matters if everything before it is accepted, so tokens deep in the block are worth less when confidence is low. Adaptive verification uses these estimates to verify the full block of 8 tokens when compute is cheap, and trim the low confidence tail under heavy load. This makes the effective speculation size adaptive to load rather than fixed at 8.

Install from the experimental branch VLLM_USE_PRECOMPILED=1 uv pip install "git+https://github.com/vllm-project/vllm.git@92eb82dbb275d7eb214b65100faec54b14102e22" --torch-backend=auto and add --speculative_config.enable_adaptive_verification true \ to your command to enable adaptive verification for further speedup. The plots below were generated on 8xB300 GPUs.

Deployment

vLLM Serving

The following example uses the BF16 checkpoint with the DSpark speculator and FlashInfer for the Mamba backend:

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve \
  nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.89 \
  --prefix-match-unit 64 \
  --block-size 64 \
  --compilation_config.max_cudagraph_capture_size 192 \
  --enable-expert-parallel \
  --max-num-seqs 128 \
  --trust-remote-code \
  --max-model-len 114688 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --async-scheduling \
  --speculative_config.method dspark \
  --speculative_config.model RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.dspark \
  --speculative_config.num_speculative_tokens 8 \
  --speculative_config.draft_tensor_parallel_size 1 \
  --speculative_config.draft_sample_method greedy \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --chat-template-content-format string \
  --mamba-cache-mode align \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3 \
  --host 127.0.0.1

Training Details

Data preparation produced 1,835,220 training examples and 653 validation examples, with sequences up to 8,192 tokens. Hidden-state extraction used four GPUs with vLLM tensor-parallel size 4, while four DDP ranks trained the draft model. The final checkpoint is trained for two epochs. It used the full verifier vocabulary, sample_from_anchor=true, and up to 1,024 anchors per sequence.

Key training configuration

train:
  speculator_type: dspark
  seed: 42
  verifier:
    verifier_name_or_path:
  data:
    total_seq_len: 8192
    train_data_ratio: 0.9996444470832133
    max_anchors: 1024
  loss:
    loss_fn: '{"ce":0.1,"tv":0.9}'
  optimizer:
    optimizer: muon
    lr: 0.003
    muon_lr: 0.003
  trainer:
    epochs: 2
    checkpoint_freq: 0.05
    fsdp_shard: false
  dflash:
    block_size: 8
    sample_from_anchor: true

Hidden-state extraction used vLLM 0.31.0 with data-parallel size 1, tensor-parallel size 4, and auxiliary target layer IDs 17 37 59 79 87. Training used Speculators commit 55ebbd5.