1. Introduction
Nanbeige4.2-3B-Base is a new base model in the Nanbeige4 series, trained from scratch with improvements to both model architecture and training data. It delivers substantial gains over the previous generation and outperforms opensource base models of comparable scale, including Qwen3.5-4B and Gemma4-E4B, across a broad range of benchmarks. Key features include:
Looped Architecture: Adopts a Looped Transformer architecture, in which hidden states are fed back into the same Transformer layers after completing a bottom-to-top pass. The reuse schema significantly increases the model capacity without increasing the number of parameters.
Enhanced Training Data Recipe: Expands the total amount of training data from 23T to 28T tokens, with more aggressive upsampling of math, code, STEM, and Synthetic-QA data to strengthen reasoning and coding capabilities.
The accompanying modeling_nanbeige.py also includes our latest architectural improvements, including LoopSplit, mHC with depth attention, and concatenated n-gram embeddings. These features have been incorporated into Nanbeige4.5, whose training is underway for release later in 2026.
2. Model Performance
We compare Nanbeige4.2-3B-Base with other base models of comparable parameter scale on a diverse benchmark suite covering mathematical reasoning, general reasoning, code generation, and knowledge-intensive tasks:
| Model | Parameters | Benchmarks | ||||||
|---|---|---|---|---|---|---|---|---|
| Total Params | Non-emb. Params | GSM8K | BBH | MBPP | MMLU-Pro | SuperGPQA | GPQA | |
| Gemma4-E4B-Base | 8B | 4B | 61.8 | 62.5 | 53.5 | 37.6 | 23.3 | 27.5 |
| Qwen3.5-4B-Base | 5B | 4B | 84.4 | 79.1 | 57.1 | 51.8 | 32.1 | 43.1 |
| Nanbeige4.1-3B-Base | 4B | 3B | 85.9 | 70.7 | 60.7 | 47.6 | 24.8 | 36.2 |
| Nanbeige4.2-3B-Base | 4B | 3B | 92.7 | 81.6 | 67.6 | 63.8 | 35.2 | 53.3 |
The results demonstrate that Nanbeige4.2-3B-Base consistently outperforms Gemma4-E4B-Base, Qwen3.5-4B-Base, and our previous Nanbeige4.1-3B-Base across the benchmark suite.
3. Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
'Nanbeige/Nanbeige4.2-3B-Base',
use_fast=False,
trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
'Nanbeige/Nanbeige4.2-3B-Base',
torch_dtype='auto',
device_map='auto',
trust_remote_code=True
)
prompt = "The capital of China is"
input_ids = tokenizer(prompt, return_tensors='pt').input_ids
output_ids = model.generate(input_ids.to('cuda'))
resp = tokenizer.decode(output_ids[0][len(input_ids[0]):], skip_special_tokens=True)
print(resp)
4. Limitations
While we place great emphasis on model safety throughout the training process, the model may still generate unexpected or inappropriate outputs due to its probabilistic nature. Such outputs may include inaccurate information, bias, discrimination, or other harmful content. Please do not propagate such content. We do not assume responsibility for consequences of disseminating inappropriate information.
5. Citation
If you find our model useful or would like to use it in your own work, please cite as follows:
@article{lab2026nanbeige4,
title={Nanbeige4. 2-3B: Unlocking Agentic Capabilities in a Compact Mode},
author={Lab, Nanbeige and Yang, Chen and Huang, Chengrui and Lan, Fufeng and Chen, Hanhui and Zhou, Hao and Song, Huatong and Cao, Jiaqi and Zhu, Jiaying and Niu, Jinlin and others},
journal={arXiv preprint arXiv:2607.22083},
year={2026}
}
6. Contact
If you have any questions, please open an issue in this repository or contact us at nanbeige@kanzhun.com.