Nemotron-H-4B-Base-8K

Model Developer: NVIDIA

Model Dates:

October 2024 - March 2025

Data Freshness:

September 2024

The pretraining data has a cutoff date of September 2024.

Model Overview

NVIDIA Nemotron-H-4B-Base-8K is a large language model (LLM) developed by NVIDIA, designed as a completion model for a given piece of text. It uses a hybrid model architecture that consists primarily of Mamba-2 and MLP layers combined with just four Attention layers. The model is pruned and distilled from Nemotron-H-8B-Base-8K using 380B tokens, and features an 8K context length. The supported languages include: English, German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, and Chinese.

For best performance on a given task, users are encouraged to customize the model using the NeMo Framework suite of customization tools, including Parameter-Efficient Fine-Tuning (P-tuning, Adapters, LoRA, and more), and Model Alignment (SFT, SteerLM, RLHF, and more) using NeMo-Aligner.

The model was pruned and distilled from Nemotron-H-Base-8K using our hybrid language model compression technique and then fine-tuned into Nemotron-H-4B-Instruct-128K. For more details, please refer to the paper.

The paper has been accepted for publication at NeurIPS 2025.

This model is for research and development only.

License/Terms of Use

GOVERNING TERMS: Use of this model is governed by the NVIDIA Internal Scientific Research and Development Model License

Model Architecture

• Architecture Type: Transformer
• Network Architecture: Nemotron-Hybrid

This model has 4B of model parameters.

Deployment Geography: Global

Use Case: This model is intended for developers and researchers building LLMs

Release Date:

Huggingface: 10/23/2025 via https://huggingface.co/

Input

• Input Type(s): Text
• Input Format(s): String
• Input Parameters: One-Dimensional (1D): Sequences
• Other Properties Related to Input: Context length up to 8K. Supported languages include German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, Chinese and English.

Output

• Output Type(s): Text
• Output Format: String
• Output Parameters: One-Dimensional (1D): Sequences

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

• Runtime Engine(s): NeMo 24.12
• Supported Hardware Microarchitecture Compatibility: NVIDIA H100-80GB, NVIDIA A100
• Operating System(s): Linux

Model Version

• v1.0

References

[2504.11409] Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning

Prompt Format

As this is a base model, no explicit prompt format is recommended or required.

Training, Testing, and Evaluation Datasets

#Training & Testing Datasets: The training corpus for Nemotron-H-4B-Base-8K consists of English and multilingual text (German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, Chinese and English), as well as code. Our sources cover a variety of document types such as: webpages, dialogue, articles, and other written materials. This model was also improved using synthetic data from Qwen (Built with Qwen). The corpus spans domains including legal, math, science, finance, and more. We also include a small portion of question-answering, and alignment style data to improve model accuracies.

Data Collection for Training & Testing Datasets: Hybrid: Automated, Human, Synthetic

Data Labeling for Training & Testing Datasets: Hybrid: Automated, Human, Synthetic

Evaluation Datasets

We used the datasets listed in the next section to evaluate the model.

Data Collection for Training Datasets: Hybrid: Automated, Human, Synthetic

Data Labeling for Training Datasets: Hybrid: Automated, Human, Synthetic

Reasoning Evaluations:

ARC (Ai2 reasoning challenge)-Challenge - The challenge set of questions from a benchmark that contains grade-school level, multiple-choice science questions to assess question answering ability of language models. Dataset

Hellaswag - Tests the ability of a language model to correctly finish the provided context from a choice of possible options. Dataset

Winogrande - Tests the ability to choose the right option for a given sentence which requires commonsense reasoning. Dataset

CommonsenseQA - A multiple-choice question answering dataset that requires different type of commonsense knowledge to predict the correct answers. Dataset

Coding Evaluations:

MBPP (Mostly Basic Python Programming Problems) - Evaluates ability to generate solutions for Python programming tasks. Dataset

MBPP+ - Extended version of MBPP with additional validation. Dataset

HumanEval - Tests code generation and completion abilities in Python. Dataset

Math Evaluations:

GSM8K (Grade School Math 8K) - Evaluates grade school level mathematical word problem solving. Dataset

General Evaluations:

MMLU - Tests knowledge across 57 subjects including science, humanities, math and more. Dataset

Potential Known Risks for Usage

The model was trained on data that contains toxic language, unsafe content, and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. The model demonstrates weakness to alignment-breaking attacks. Users are advised to deploy language model guardrails alongside this model to prevent potentially harmful outputs. The model may generate answers that are inaccurate, omit key information, or include irrelevant or redundant text.

Inference

• Engine: NeMo
• Test Hardware NVIDIA H100-80GB

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Responsible Use Guide available at http://nvidia.com/nemotron-responsible-use.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Sample code

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the tokenizer and model
tokenizer  = AutoTokenizer.from_pretrained("nvidia/Nemotron-H-4B-Base-8K", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("nvidia/Nemotron-H-4B-Base-8K", torch_dtype=torch.bfloat16, trust_remote_code=True).cuda()

prompt = "When was NVIDIA founded?"

outputs = model.generate(**tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device))
print(tokenizer.decode(outputs[0]))