🤗 Hugging Face | 🤖 ModelScope | 🐙 OpenRouter
Introduction
We have open-sourced the Ling-3.0 series, our most efficient language foundation model family to date. To support research and community-driven innovation, we are releasing a collection of checkpoints during the training process as following:
Model
Pre-trained
Mid-trained
Merged (i.e., WSM)
Ling-3.0-tiny
Ling-3.0-tiny-base-30T
Ling-3.0-tiny-base-midtrain
Ling-3.0-tiny-base
Ling-3.0-flash
Ling-3.0-flash-base-30T
Ling-3.0-flash-base-midtrain
Ling-3.0-flash-base
These checkpoints correspond to different stages of the training process:
- Pretrained checkpoint have completed large-scale pretraining but have not undergone mid-training, WSM merging (or learning-rate decay), or post-training.
- Mid-trained checkpoint have completed mid-training but have not undergone WSM merging (or learning-rate decay) or post-training.
- Merged checkpoints have undergone WSM merging (or learning-rate decay) based on the mid-training checkpoints but have not undergone post-training.
These checkpoints are released to support continued pretraining, fine-tuning, and further research. For the post-trained model, please see and see Ling-3.0-tiny and Ling-3.0-flash.
Model Overview
Key features
- Highly sparse (1/64) MoE architecture: 512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token;
- Native hybrid linear attention: Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
- Warmup-Stable and Merge: We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
- Scale Seamlessly: Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
Model Type
Base (final checkpoint of mid-training)
ArchitectureHybrid-linear MoE
Parameter ScaleTotoal 124B, Activated 5.1B (Non-emb)
Transformer Layers35 KDA + 7 Gated MLA (5:1)
Number of Dense Layers2
Number of Routed Experts512
Number of Shared Experts1
Number of Activated Experts8
Attention Heads32
Hidden Size2560
Expert Intermediate Size768
Dense Intermediate Size6144
Vocabulary Size157,184
Base Model Evaluation
To systematically assess the capabilities of the base model, we use a self-built comprehensive benchmark suite covering several key domains, including mathematics, coding, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e., Ling-3.0-flash-base, is compared below:
.benchmark-table {
border-collapse: collapse;
margin: 0 auto;
}
.benchmark-table th,
.benchmark-table td {
padding: 0.55rem 0.8rem;
text-align: center !important;
vertical-align: middle;
}
.benchmark-table .ling {
background-color: #e7f6ff;
}
.benchmark-table thead .ling {
color: #1677ff;
}
.benchmark-table thead tr:first-child .ling {
border-radius: 28px 28px 0 0;
}
.benchmark-table tbody tr:last-child .ling {
border-radius: 0 0 28px 28px;
}
.benchmark-table .section-start > * {
border-top: 2px solid #d0d7de;
}
.benchmark-table ins {
text-decoration-thickness: 1px;
text-underline-offset: 0.12em;
}
.benchmark-table .table-footnote {
padding-top: 0.85rem;
text-align: left !important;
color: #57606a;
font-size: 0.9em;
}
Domain
Benchmark
Shot Config
Ling-3.0-flash-base
Ling-2.5-flash-base
NVIDIA-Nemotron-3-Super-base
Step-3.5-Flash-base
Hy3-preview-base
MiMo-V2.5-base
124B A5.1B
104B A7.4B
120B A12B
196B A11B
295B A21B
310B A15B
Knowledge
CCPM(EM)
0-shot
90.00
78.38
80.77
83.90
88.64
88.05
ARC-C(EM)
0-shot
96.27
95.59
96.27
92.88
94.58
95.59
AGIEval(Acc)
0-shot
77.50
72.39
67.08
74.27
77.61
71.84
SimpleQA-Verified(Acc)
5-shot
26.90
15.90
21.20
27.80
24.20
19.10
MMLU-Pro (EM)
5-shot
67.74
61.36
63.89
63.65
66.44
64.90
CEval(EM)
5-shot
89.76
87.86
78.12
89.36
90.74
88.60
Code
HumanEval-Plus (Pass@1)
0-shot
81.10
80.49
78.05
78.05
79.27
76.22
CruxEval(Pass@1)
1-shot
78.50
76.50
72.44
71.69
82.94
70.06
MultiPL-E(Pass@1)
1-shot
70.79
68.07
57.28
63.21
45.77
53.57
LiveCodeBench1 (Pass@1)
1-shot
40.09
33.04
38.99
35.68
35.90
35.50
BigCodeBench(Pass@1)
0-shot
52.19
50.35
25.53
49.74
52.02
50.18
FullStackBench(Pass@1)
3-shot
51.16
48.43
49.61
50.86
54.95
50.39
LCBench2 (Pass@1)
3-shot
57.76
57.51
54.22
49.67
46.94
45.12
Math
MATH500(Acc)
4-shot
79.00
74.00
70.80
62.80
60.60
70.00
OlympiadBench (Acc)
3-shot
47.89
39.31
54.52
34.34
41.27
35.09
TheoremQA(Acc)
5-shot
61.10
57.95
72.05
56.03
60.96
51.23
OmniMath (Acc)
3-shot
47.65
36.20
38.55
28.46
33.33
27.03
Reasoning
CommonSenseQA (EM)
5-shot
89.93
87.55
86.24
88.62
86.98
84.60
BBH (EM)
3-shot
89.17
84.72
89.00
87.19
76.00
86.22
Long-context
LongBench (Acc)
0-shot
52.62
42.81
31.05
21.47
20.58
29.57
LEval (Acc)
0-shot
83.24
76.89
58.65
60.30
66.29
73.29
Note:
1 LiveCodeBench (2408-2505)
2 LCBench (2301-2502)
Intended Use
Recommended use cases:
- Continued pre-training
- Mid-training
- Supervised fine-tuning for domain adaptation
- Preference optimization and RL post-training Distillation research
- Long-context and MoE systems research
Not recommended as-is for:
- Direct end-user chat deployment
- Safety-critical applications without additional alignment and evaluation
- Production use without post-training and task-specific validation
Usage
For fine-tuning examples, please refer to our ling-cookbook.
FAQ
If you have any question, please feel free to add a discussion.
License
This model is released under the MIT License.