Model Overview

Description:

The NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash model is the GGUF (a binary model file format used by llama.cpp) version of NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash, the DFlash speculative decoding draft checkpoint for NVIDIA-Nemotron-3.5-Bolt-E4B, a reasoning language model post-trained by NVIDIA from Google's Gemma 4 E4B IT. It is converted to GGUF in BF16 (bfloat16) for use with llama.cpp and must be loaded together with the GGUF version of the target model, Nemotron-3.5-Bolt-E4B-4GB. The draft model does not include token embeddings or an output head, and uses those of the target model at runtime. The model is intended for lower-latency local serving of NVIDIA-Nemotron-3.5-Bolt-E4B on workstation GPUs.

This model is ready for commercial or non-commercial use.

License/Terms of Use:

GOVERNING DOWNLOAD TERMS: Use of the model is governed by the OpenMDW License Agreement, version 1.1. ADDITIONAL INFORMATION: Apache License 2.0 (Gemma 4 E4B IT).

Deployment Geography:

Global

Use Case:

Developers running NVIDIA-Nemotron-3.5-Bolt-E4B locally with llama.cpp for reasoning, chat, tool use, and agentic workflows that benefit from lower-latency speculative decoding on workstation GPUs. This release is intended for DFlash-assisted serving together with NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF rather than as a standalone model checkpoint.

Release Date:

Hugging Face 10/23/2026 via https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash

References

• NVIDIA Model Optimizer: https://github.com/NVIDIA/Model-Optimizer
• llama.cpp: https://github.com/ggml-org/llama.cpp
• NVIDIA-Nemotron-3.5-Bolt-E4B-DFlash model card
• Nemotron-3.5-Bolt-E4B-BF16 model card
• Nemotron-3.5-Bolt-E4B-4GB model card
• Gemma 4 E4B IT model card
• OpenMDW License Agreement, version 1.1
• DFlash: Block Diffusion for Flash Speculative Decoding

Model Architecture:

Architecture Type: Transformer
Network Architecture: Other - DFlash draft model (five Qwen3-style decoder layers)
Number of Model Parameters: 360M (360,480,000), stored in BF16 in a single GGUF file of 736,812,096 bytes (736.81 MB / 702.68 MiB). The draft model shares the token embeddings and output head of the target model, so they are not included in this checkpoint.

For more information about the underlying model's architecture, please see Nemotron-3.5-Bolt-E4B-BF16.

Input:

Input Type(s): Text
Input Format(s): String
Input Parameters: One-Dimensional (1D)
Other Properties Related to Input: Maximum context length of 131,072 tokens, set in llama.cpp with -c. The draft model attends to the most recent 1,024 tokens through its sliding-window attention.

Output:

Output Type(s): Text
Output Format: String
Output Parameters: One-Dimensional (1D): Sequences
Other Properties Related to Output: The draft model proposes up to 7 candidate tokens per decoding step, and the target model verifies them, so the final output is generated by the target model.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Supported Runtime Engine(s):

• llama.cpp b11808

Supported Hardware Microarchitecture Compatibility:

• NVIDIA RTX 5090

Preferred Operating System(s):

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

v1.0: NVIDIA-Nemotron-3.5-Bolt-E4B-DFlash-GGUF is the BF16 GGUF version of NVIDIA-Nemotron-3.5-Bolt-E4B-DFlash, which was trained with NVIDIA Model Optimizer 0.48.0. It was converted with llama.cpp convert_hf_to_gguf.py at commit [4912ac2]. To integrate it into an AI system, load it with -md alongside NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF in llama-server (see Usage); applications then query the llama-server endpoint as usual.

Training and Evaluation Datasets:

Training Dataset:

For more information about the underlying NVIDIA-Nemotron-3.5-Bolt-E4B model, please visit the model card - Nemotron-3.5-Bolt-E4B-BF16.

Link: Nemotron-Post-Training-Dataset-v2, only prompts from the datasets were used for data synthesis (the original responses from GPT were not used), which is then used to train the DFlash modules.
Data Modality: Text
Text Training Data Size: 1 Billion to 10 Trillion Tokens
Data Collection Method by dataset: Hybrid: Automated, Synthetic
Labeling Method by dataset: Hybrid: Automated, Synthetic
Properties: About 3.7 million multilingual text samples featuring prompts spanning math, code, STEM, and conversational topics. Each sample includes a synthetic response generated by the target model; the corpus mixes responses generated with thinking enabled, which include their reasoning traces, and with thinking disabled.

Testing Dataset:

Link: SPEED-Bench, MT-Bench
Data Collection Method by dataset: Hybrid: Automated, Manually-Collected, Synthetic
Labeling Method by dataset: Hybrid: Automated, Manually-Labeled, Synthetic
Properties: The SPEED-Bench qualitative split (880 text prompts in 11 task categories, mostly English) and MT-Bench (80 English two-turn text questions in 8 categories) were used during development to measure acceptance length and select checkpoints.

Evaluation Dataset:

The model is evaluated on the SPEED-Bench and MT-Bench datasets for speculative decoding benchmarking.

Data Collection Method by dataset: Hybrid: Automated, Manually-Collected, Synthetic
Labeling Method by dataset: Hybrid: Automated, Manually-Labeled, Synthetic
Properties: All prompts are text and mostly in English. The SPEED-Bench qualitative split has 880 prompts in 11 task categories (such as coding, math, reasoning, and writing, plus a multilingual category covering 23 languages); MT-Bench has 80 two-turn questions in 8 categories. Acceptance length over the target model's responses measures the quality of the draft model.

Inference:

Acceleration Engine: llama.cpp
Test Hardware: NVIDIA RTX 5090

DFlash Speculative Decoding

NVIDIA-Nemotron-3.5-Bolt-E4B-DFlash-GGUF is a DFlash draft model for NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF. In each decoding step, the draft model takes the last verified token together with hidden states captured from five layers of the target model, and proposes the next 7 tokens of an 8-token block in a single non-causal forward pass. The target model verifies the proposed tokens in one forward pass and accepts the longest valid prefix, so more than one token can be generated per step. The mean number of tokens generated per step is the acceptance length, with a ceiling of 8.

The draft model was trained on a mix of thinking-enabled and thinking-disabled conversations. llama-server enables thinking by default (--reasoning auto), which matches the setting used for the acceptance lengths reported below.

Usage

To serve the model with llama.cpp b11808, load the target GGUF with -m and this draft GGUF with -md:

# [VERIFY: file names, quantization level, and flags after the llama.cpp test]
llama-server \
  -m NVIDIA-Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf \
  -md NVIDIA-Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 7 \
  -ngl 99 \
  -fa on \
  --jinja \
  --port 8000

Evaluation

The acceptance lengths below were measured with the Hugging Face checkpoint of this draft model in llama.cpp, against the target model with thinking enabled, 4,096 output tokens, and concurrency 1 on one NVIDIA RTX 5090 GPU. This GGUF file keeps the same BF16 weights; acceptance lengths in vLLM may differ slightly.

Acceptance length on MT-Bench with draft length 7:

Acceptance length on SPEED-Bench (qualitative split) with draft length 7:

Model Limitations:

The base model may generate inaccurate, incomplete, or otherwise undesirable responses, even when prompts are benign. It may reflect biases or content artifacts present in its training data, and output quality can vary by domain, language, reasoning depth, and context length. Developers should add application-specific safeguards and evaluate the system in the intended deployment environment before production use. The speedup from the draft model depends on the serving configuration and the quantization of the target model.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

SUBCARDS:

Explainability

Bias

Safety & Security

Privacy

The datasets used to train NVIDIA-Nemotron-3.5-Bolt-E4B-DFlash-GGUF did not include personal data.

NVIDIA collected and used the training data in compliance with applicable data protection and privacy laws. The datasets used to train this model were reviewed before release. Please review NVIDIA’s Privacy Policy for more information.