Nemotron 3.5 Parse

Description:

Nemotron 3.5 Parse is a document image parser that produces structured, machine-readable text with semantic layout classes, bounding boxes, and reading order. It combines a C-RADIOv4-SO400M vision encoder with a 10-layer mBART decoder that consumes the visual sequence as a causal prefix. The release uses dynamic rectangular NaViT preprocessing, content-normalized coordinates, grouped-query self-attention (16 query heads and 8 key/value heads), a 72,505-token multilingual tokenizer, and chart-aware output through the <class_Chart> token. It is intended for document understanding, information retrieval, data extraction, and multimodal data-curation workflows.

Nemotron 3.5 Parse was developed by NVIDIA as a part of Nemotron.

This model is ready for commercial or non-commercial use.

Hosted inference

• Try the API on NVIDIA Build
• Deploy on Baseten

Benchmark highlights

Detailed benchmarks: ParseBench · IndicVisionBench · MOSCAR.

ParseBench results use triple pass with region union on 2,078 unique pages. See pipeline details, reproduction instructions, and evaluation results.

Additional OCR highlights (scores on a 0–100 scale):

• IndicVisionBench OCR: 83.73 ANLS-char, evaluated on 876 test pages across 10 languages. Per-language results.
• MOSCAR: 99.51 character bag F1, evaluated on 8,111 simply rendered synthetic pages covering 163 language/script entries. Per-script results and full per-language results.

MOSCAR uses simple synthetic renderings of text in each language to test basic OCR capability. These results do not guarantee accuracy on complex scanned documents, handwriting, degraded images, or other real-world variations. Character bag F1 measures character overlap without considering character order.

License/Terms of Use:

Governing Terms: Your use of this model is governed by the OpenMDW License Agreement, version 1.1. Use of the tokenizer included in this model is governed by the MIT license and tokenizer attribution. See third-party notices for separately installed dependencies.

Tokenizer source and modifications are documented in TOKENIZER_ATTRIBUTION.md.

The portable ParseBench pipeline has its own license and third-party notices, including the separately installed benchmark environment. See the component inventory and package build evidence for exact versions and digests.

Deployment Geography:

Global

Use Case:

Nemotron 3.5 Parse is designed for developers and teams building document intelligence, retrieval-augmented generation (RAG), curator, extractor, and agentic AI applications. It can convert scanned or rendered PDFs, presentation slides, forms, reports, tables, and mixed-content document pages into structured outputs for downstream indexing, retrieval, analytics, model training-data creation, and human-in-the-loop review.

Release Date:

Hugging Face 10/17/2026 via https://huggingface.co/nvidia/Nemotron-3.5-Parse

References:

• Hugging Face Transformers mBART documentation
  
• NVIDIA C-RADIOv4-SO400M
  

Model Architecture:

Architecture Type: Transformer

Network Architecture:

• Vision Encoder: NVIDIA C-RADIOv4-SO400M (vit_so400m_patch16_224) with dynamic rectangular NaViT preprocessing
  
• Adapter Layer: 1D convolutions and normalization layers that compress the vision latent sequence before decoding
  
• Decoder: 10-layer mBART causal visual-prefix decoder with 16 query heads and 8 key/value heads
  
• Auxiliary Prediction Head: One trained one-step multi-token-prediction (MTP) head is included in model.safetensors. Transformers uses the normal decoder for generation; the vLLM plugin enables the bundled head for one-token speculative decoding.
  
• Tokenizer: 72,505 entries with multilingual and task/control tokens including <predict_bbox>, <predict_classes>, <predict_text_in_pic>, and <class_Chart>; the decoder embedding is padded to 72,512 rows. Use of the tokenizer included in this model is governed by the MIT license and tokenizer attribution.
  
• Number of Parameters: 630,800,496 unique trainable parameters (approximately 0.631B), including the MTP head; shared decoder embedding/output weights are counted once
  

Input(s):

Input Type(s): Image, Text

Input Format(s):

• Image: Red, Green, Blue (RGB)
  
• Text: Prompt string
  

Input Parameters:

• Image: Two-Dimensional (2D)
  
• Text: One-Dimensional (1D)
  

Other Properties Related to Input:

• Recommended maximum input resolution (Width, Height): 1664, 2048
  
• Channel Count: 3
  
• Prompt format: one task prompt composed from the supported control tokens. The default prompt extracts bounding boxes, semantic classes, and text in markdown format: </s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>.
  

Output(s):

Output Type(s): Text

Output Format(s): String

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output: Nemotron 3.5 Parse returns a string encoding document text, semantic element classes, and content-normalized bounding boxes. Scale x coordinates by the evaluated image width and y coordinates by its height. Postprocessing utilities can convert table or chart-associated text into LaTeX, HTML, markdown, JSON, or CSV where supported.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA hardware and software frameworks, the model achieves faster training and inference times compared to CPU-only solutions.

Quick Start

Transformers installation

Use Python 3.12 and a fresh environment:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install "transformers==5.14.1" "torchvision==0.26.0" "timm==1.0.22"

Run hf auth login if repository access requires authentication. Transformers downloads the model weights, processor, and custom model code automatically. No Nemotron package installation is needed for Transformers inference.

Usage example

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor, GenerationConfig

model_path = "nvidia/Nemotron-3.5-Parse"
device = "cuda:0"

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    dtype=torch.bfloat16
).to(device).eval()
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image = Image.open("path/to/your/image.jpg").convert("RGB")
task_prompt = "</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>"

inputs = processor(images=[image], text=task_prompt, return_tensors="pt", add_special_tokens=False).to(device)

generation_config = GenerationConfig.from_pretrained(model_path)
generation_config.max_new_tokens = 8744

with torch.inference_mode():
    outputs = model.generate(**inputs, generation_config=generation_config)

generated_text = processor.batch_decode(outputs, skip_special_tokens=False)[0]
print(generated_text)

Postprocessing

The shared helpers render a complete document as Markdown with HTML tables, and return structured blocks with normalized boxes and optional pixel boxes. They preserve model reading order, table spans, blank cells, and in-cell line breaks. They also decode rich table formatting, remove literal duplicate elements, repair empty serialization prefixes, and normalize existing list bullets and discretionary soft hyphens in text. Code and math are protected from typography cleanup.

Download the helpers and their shared implementation into your working directory (no Nemotron package installation is needed for this path):

python -m pip install "beautifulsoup4==4.13.3"
hf download nvidia/Nemotron-3.5-Parse \
  --include 'postprocessing.py' 'latex2html.py' 'nemotron_parse/**' --local-dir .

Continue from the Transformers example above:

from pathlib import Path
from PIL import ImageDraw
from postprocessing import postprocess_document

document = postprocess_document(generated_text, image_size=image.size)
Path("prediction.md").write_text(document["markdown"], encoding="utf-8")
draw = ImageDraw.Draw(image)
for block in document["blocks"]:
    draw.rectangle(block["bbox_pixels"], outline="red", width=2)
image.save("prediction.png")

The existing extract_classes_bboxes, transform_bbox_to_original, and postprocess_text functions remain available. Supported table_format values: latex | HTML | markdown | json | json_hierarchical | csv. HTML and hierarchical JSON retain merged-cell structure; flat Markdown, CSV, and record JSON flatten spans. Headerless tables use neutral column names in flat exports, retaining every data row.

Explicit chart titles, subtitles, and panel labels are preserved with their tables. Header inference uses predicted spans and existing bold markup; it can be disabled. Numeric value/percentage column splitting and geometric chart-caption association are enabled by default. Both preserve existing predicted values/text and can be disabled. See postprocessing examples for controls and examples.

Inference with vLLM

vLLM requires the Nemotron plugin for this architecture and its MTP head. The plugin accepts stable vLLM releases >=0.25.0,<0.33 (0.25.x through 0.32.x). In a fresh Python 3.12 environment, download the repository and install the serving extras:

python -m pip install "huggingface-hub>=1.24.0,<2"
hf download nvidia/Nemotron-3.5-Parse --local-dir nemotron-parse
cd nemotron-parse
python -m pip install ".[vllm,client]" "vllm>=0.25.0,<0.33"

To select a specific vLLM release within this range, replace the final argument with your desired version (for example, "vllm==0.32.0"). Let pip resolve the compatible Torch, torchvision, and Transformers dependencies declared by the package.

One-token MTP and the compatible FA3 scheduler are enabled automatically.

vLLM Inference example

Option 1: Python inference with MTP

from PIL import Image
from vllm import LLM, SamplingParams

def main():
    llm = LLM(
        model=".",
        dtype="bfloat16",
        trust_remote_code=True,
        distributed_executor_backend="uni",
        max_model_len=16384,
        max_num_seqs=64,
        max_num_batched_tokens=16384,
        limit_mm_per_prompt={"image": 1},
        enable_prefix_caching=False,
        attention_config={"flash_attn_version": 3},
    )
    sampling_params = SamplingParams(
        temperature=0,
        top_k=1,
        repetition_penalty=1.1,
        max_tokens=8744,
        skip_special_tokens=False,
    )
    image = Image.open("<YOUR-IMAGE-PATH>").convert("RGB")
    outputs = llm.generate([{
        "prompt": "<pad>\n</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>",
        "multi_modal_data": {"image": image},
    }], sampling_params, tokenization_kwargs={"add_special_tokens": False})
    print(outputs[0].outputs[0].text)

if __name__ == "__main__":
    main()

The offline prompt includes the image placeholder <pad>. Automatic special tokens are disabled because the task prompt already contains its control tokens.

Option 2: vLLM serve with MTP

vllm serve . \
    --served-model-name nemotron-parse \
    --dtype bfloat16 \
    --distributed-executor-backend uni \
    --tensor-parallel-size 1 \
    --max-model-len 16384 \
    --max-num-seqs 192 \
    --max-num-batched-tokens 16384 \
    --no-enable-prefix-caching \
    --limit-mm-per-prompt '{"image":1}' \
    --attention-config '{"flash_attn_version":3}' \
    --trust-remote-code \
    --port 8000 \
    --chat-template chat_template.jinja

Then run inference through the OpenAI-compatible API:

import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

with open("<YOUR-IMAGE-PATH>", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode("utf-8")

prompt_text = "</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>"

resp = client.chat.completions.create(
    model="nemotron-parse",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": prompt_text,
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{img_b64}",
                    },
                },
            ],
        }
    ],
    max_tokens=8744,
    temperature=0.0,
    extra_body={
        "repetition_penalty": 1.1,
        "top_k": 1,
        "skip_special_tokens": False,
    },
)
print(resp.choices[0].message.content)

For Docker deployment and optional generation controls, see the serving guide.

Software Integration:

Runtime Engine(s):

• Transformers
  
• vLLM
  

Reference Hardware: NVIDIA Hopper H100, BF16. The FA3/MTP speed profile above is measured on H100.

Supported Operating System(s):

• Linux
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels is essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

Nemotron 3.5 Parse · October 17, 2026 release

This AI model can be embedded as an Application Programming Interface (API) call through the documented Transformers or vLLM serving interfaces.

Training, Testing, and Evaluation Datasets:

Training Dataset

Data Modality:

• Image
  
• Text
  

Image Training Data Size:

• 1 Million to 1 Billion Images
  

Text Training Data Size:

• 1 Billion to 10 Trillion Tokens
  

Data Collection Method by dataset:

• Automated, Manually-Collected, Synthetic
  

Labeling Method by dataset: Automated, Manually-Labeled, Synthetic
Labeling Details: Labels are produced through a combination of automated OCR and layout annotation, manual annotation, and synthetic generation. The combination varies by constituent dataset.

Properties: The training set contains millions of image-text items aggregated across large document, table, and layout datasets. The data consists of document-page and table images paired with OCR text, bounding boxes, and layout labels. Sources include rendered digital documents, scientific papers, PDFs, Wikipedia-style pages, and synthetic document, table, word, and character renderings. Annotations come from OCR and layout models, third-party OCR services, synthetic-generation pipelines, and human labeling.

Testing Dataset

Data Collection Method by dataset: Automated, Manually-Collected, Synthetic
Collection Details: Sources combine automated collection, manually collected material, and synthetic generation; methods vary by constituent dataset.

Labeling Method by dataset: Automated, Manually-Labeled, Synthetic
Labeling Details: Labels are produced through a combination of automated OCR and layout annotation, manual annotation, and synthetic generation. The combination varies by constituent dataset.

Properties: Tested on internal and public document-understanding benchmarks using document-page images and reference text, layout classes, bounding boxes, and structure annotations. Coverage includes OCR quality, layout structure, table parsing, reading order, and visual grounding across document types and languages.

Evaluation Dataset

Data Collection Method by dataset: Automated, Manually-Collected, Synthetic
Collection Details: Sources combine automated collection, manually collected material, and synthetic generation; methods vary by constituent dataset.

Labeling Method by dataset: Automated, Manually-Labeled, Synthetic
Labeling Details: Labels are produced through a combination of automated OCR and layout annotation, manual annotation, and synthetic generation. The combination varies by constituent dataset.

Properties: Evaluated on internal and public document-understanding benchmarks covering OCR quality, layout structure, table parsing, reading order, and visual grounding. The ParseBench reference evaluation covers 2,078 unique document pages. Additional benchmarks cover Indic-language OCR and multilingual synthetic OCR across the scripts listed below.

Benchmark coverage:

• ParseBench evaluates document parsing across text fidelity, semantic formatting, tables, charts, and visual grounding with layout boxes/classes.
  
• IndicVisionBench evaluates OCR quality for Indic-language pages, including Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu.
  
• MOSCAR evaluates multilingual synthetic OCR across broad script coverage, including Latin, Arabic, Cyrillic, Han, Hangul, Japanese, Indic scripts, Hebrew, Thai, Greek, and others.
  

Evaluation Results

ParseBench

• Single pass: parse the full page once.
• Double pass: parse the page, then refine Table/Chart crops selected from its predictions.
• Triple pass: add a separate detection request and refine the union of detector and parser regions. Full-page parsing and detection can overlap.

On ParseBench, double pass gains 2.18 overall points over single pass. Triple pass adds 0.28 points over double pass at about 9% lower throughput. Double pass captures most of the measured accuracy gain; the table shows the speed tradeoff.

Published overall scores with locally measured throughput:

Speeds use the same 2,078 prepared page images on one H100 per model. Timing includes all model passes and dependent crop work, excluding initial page preparation, startup, warmup, final document rendering and writes.

The refinement merge retains differing full-page/crop alternatives; the reported scores include these alternatives.

See score sources and timing details, full results, and the pipeline reproduction instructions.

IndicVisionBench OCR

OCR results on 876 test pages across 10 languages. ANLS-char and character error rate (CER) are shown as percentages. CER includes insertion errors and can exceed 100%.

MOSCAR multilingual OCR

MOSCAR uses simple synthetic renderings of text in each language to test basic OCR capability. These results do not guarantee accuracy on complex scanned documents, handwriting, degraded images, or other real-world variations. Character bag F1 measures character overlap without considering character order.

OCR results on 8,111 held-out synthetic pages covering all 163 configured language/script entries across 26 script groups. Character bag F1 (BoC F1, higher is better) and CER (lower is better) are percentages. Overall CER is computed across all reference characters, so it is not an unweighted average of the script rows.

Both comparisons use the released Parse 2.0 weights and current Parse 3.5 weights, identical page sets, greedy decoding, and the same scorer for each benchmark. Each model uses its documented default task prompt. All outputs, including those reaching the token limit, are included. See the full MOSCAR results for all 163 language/script entries, downloadable results, and evaluation settings.

Inference reference outputs and verification instructions are available in reference/.

The quick20/full500 regression references were refreshed for the released checkpoint on October 18, 2026. Full500 completed 500/500 pages without request failures and scored 74.49% overall and 83.38% table quality with the released table parser. These are lightweight regression metrics, separate from official ParseBench or OmniDocBench TEDS. Raw outputs, legacy scores and the updated scorer are included.

Inference:

Acceleration Engine: Transformers, vLLM
Test Hardware:

• NVIDIA Hopper H100
  

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content. If an input image contains people, personal health information, confidential business data, or intellectual property, the model may extract or reproduce visible text from that content.

For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy subcards.
Please report model quality, risk, security vulnerabilities, or NVIDIA AI Concerns here.

Parsing helpers

The Python helpers return Markdown, structured blocks, and raw model output. After installing the package and starting the vLLM server above:

from nemotron_parse.inference import DocumentParser

with DocumentParser() as parser:
    document = parser.parse("page.png")
print(document["markdown"])

See the postprocessing guide for pipeline options, prepared batches, high concurrency, CLI examples, and output controls.