NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16

Looking for a post-trained model? See the BF16 release for customization or the NVFP4 release for optimized deployment.

Model Overview

Model Developer: NVIDIA Corporation

Description

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 is the base vision-language checkpoint of the Nemotron 3.5 Super family. It provides a starting point for continued training, domain adaptation, and developing custom post-trained vision-language models.

The language backbone is initialized from Nemotron 3 Super Base and further trained through distillation from Nemotron 3 Ultra Base. The model underwent continued pre-training on indic languages, long-context training, and software-engineering mid-training. A vision encoder and connector extend the language backbone to visual inputs.

The model combines interleaved Mamba-2, Mixture-of-Experts (MoE), and attention layers with a C-RADIO vision encoder and a vision-language connector. It has 120B total parameters and 12B active parameters and supports context lengths up to 1M tokens. Multi-Token Prediction (MTP) enables native speculative decoding.

This model is ready for commercial or non-commercial use.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.

License/Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1.

Benchmarks

The following results evaluate the Nemotron 3.5 Super text-only base backbone, before vision-language training.

Deployment Geography: Global

Use Case

This model is intended for developers and researchers building custom vision-language models through continued pre-training, supervised fine-tuning, reinforcement learning, and distillation.

Release Date

Hugging Face - 10/15/2026 via Hugging Face

Reference(s)

• NVIDIA Nemotron 3 model family on Hugging Face
• NVIDIA Nemotron 3.5 Super VL and Lightning Technical Report
• NVIDIA Nemotron 3 Ultra Technical Report

Model Architecture

• Architecture Type: Mamba-2–Transformer hybrid Mixture of Experts (MoE) with Multi-Token Prediction (MTP)
• Network Architecture: Nemotron Hybrid MoE with 512 routed experts per MoE layer, a C-RADIO vision encoder, and a vision-language connector
• Number of Model Parameters: 120B total / 12B active

Model Design

The model combines sparse expert computation, attention, and Mamba-2 layers to support long-context processing. The vision encoder and connector turn visual inputs into representations processed by the language backbone. A shared MTP head predicts future tokens recursively for speculative decoding.

The model supports up to 1M tokens of context. The prompt, visual inputs, conversation history, tool results, and generated response share this context window. Individual deployment recipes may use a smaller context window to accommodate the available GPU memory and concurrency.

Training Methodology

Stage 1: Foundation and Multi-Token Prediction

The language backbone is initialized from Nemotron 3 Super Base. A shared MTP head is trained for 100B tokens while the backbone remains frozen, followed by 1T tokens of distillation from Nemotron 3 Ultra Base.

Stage 2: Continued Pre-Training and Mid-Training

The model undergoes 100B tokens of Indic continued pre-training, a 35B-token extension to 1M context, 67B tokens of software-engineering mid-training at 256K context, and a final 9B-token long-context recovery stage. Software-engineering data includes repositories, issue–pull-request pairs, commit diffs, build and CI logs, and agent trajectories.

Stage 3: Vision-Language Training

A C-RADIO vision encoder and connector extend the language backbone to visual inputs, including images, document pages, and video frames.

Input

• Input Type(s): Text, images, and video
• Input Format(s): Text strings, images, and sampled video frames
  • Video: mp4, up to 2 minutes.
  • Image: Red, Green, Blue (RGB) (jpeg, png)
• Input Parameters:
  • Video: Three-Dimensional (3D)
  • Image: Two-Dimensional (2D)
  • Text: One-Dimensional (1D)
• Other Properties Related to Input: Maximum context length up to 1M tokens. Documents can be supplied as page images with accompanying text.

Output

• Output Type(s): Text
• Output Format: String
• Output Parameters: One-dimensional sequences
• Other Properties Related to Output: Maximum context length up to 1M tokens.

Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

• Runtime Engine(s): PyTorch; Megatron-LM; NeMo
• Hardware:
  • NVIDIA Blackwell
  • NVIDIA Hopper
• Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

• v1.0 - GA

Training, Testing, and Evaluation Datasets

Training

Data Modality: Text, images, and video
Foundation Training Data Size: More than 20 trillion tokens
Image Training Data Size: 1 Million to 1 Billion Images
Text Training Data Size: More than 10 Trillion Tokens
Video Training Data Size: 10,000 to 1 Million Hours
Foundation Data Collection Period: 2013 to December 2025
Data Collection Method: Hybrid: automated, manually-collected, and synthetic
Labeling Method: Hybrid: automated, manually-labeled, and synthetic

The foundation corpus contains curated and synthetic text covering English, 19 other spoken languages, and 43 programming languages. Sources include webpages, dialogue, articles, code, and other written materials across math, science, legal, finance, and general knowledge domains.

For more information about the datasets used to train this model, please see the Public Summary of Training Content

Detailed dataset information

Base Pre-Training Corpus (Nemotron 3 Foundation)

The foundation of the model is trained on the Nemotron 3 corpus, comprising the following datasets from the Nemotron Pretraining Datasets collection:

Public Datasets

Crawled and Scraped from Online Sources by NVIDIA

The English Common Crawl data was downloaded from the Common Crawl Foundation (see their FAQ for details on their crawling) and includes the snapshots CC-MAIN-2013-20 through CC-MAIN-2025-13. The data was subsequently deduplicated and filtered in various ways used in the Nemotron-CC data pipeline. Additionally, we extracted data for fifteen languages from the following three Common Crawl snapshots: CC-MAIN-2024-51, CC-MAIN-2025-08, CC-MAIN-2025-18. The fifteen languages included were Arabic, Chinese, Danish, Dutch, French, German, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, and Thai. As we did not have reliable multilingual model-based quality classifiers available, we applied just heuristic filtering instead—similar to what we did for lower quality English data in the Nemotron-CC pipeline, but selectively removing some filters for some languages that did not work well. Deduplication was done in the same way as for Nemotron-CC.

The GitHub Crawl was collected using the GitHub REST API and the Amazon S3 API. Each crawl was operated in accordance with the rate limits set by its respective source, either GitHub or S3. We collect raw source code and subsequently remove any having a license which does not exist in our permissive-license set.

Private Non-publicly Accessible Datasets of Third Parties

Private Non-publicly Accessible Datasets by NVIDIA

NVIDIA-Sourced Synthetic Datasets

Additional Training Data

Testing Datasets

Data Collection Method: Hybrid: automated, manually-collected, and synthetic
Labeling Method: Hybrid: automated, manually-labeled, and synthetic
Properties: Standard benchmarks and task-specific test suites covering knowledge, math, code, commonsense reasoning, reading comprehension, multilingual understanding, and long context.

Evaluation Datasets

Data Collection Method: Hybrid: automated, manually-collected, and synthetic
Labeling Method: Hybrid: automated, manually-labeled, and synthetic
Properties: Standard benchmarks and task-specific test suites covering knowledge, math, code, commonsense reasoning, reading comprehension, multilingual understanding, and long context.

Inference

• Acceleration Engine: PyTorch

• Test Hardware:

  • NVIDIA Hopper
    • H100
    • H200
  • NVIDIA Grace Blackwell
    • GB200
    • GB300
  • NVIDIA Blackwell
    • B200
    • B300

The Base model is intended as a starting point for further training. For chat, reasoning, and tool-calling applications, use the post-trained BF16 or NVFP4 release and its deployment instructions.

Download the Base weights with the Hugging Face CLI:

hf download nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 \
  --local-dir ./models/super-vl-base-bf16

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

We advise against circumvention of any provided safety guardrails contained in the Model without a substantially similar guardrail appropriate for your use case. For more details: Safety and Explainability subcards. For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, and Privacy subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.