🤗 Hugging Face | 🤖 ModelScope | 🐙 OpenRouter
We have open-sourced the Ling-3.0 series, our most efficient language foundation model family to date. To support research and community-driven innovation, we are releasing a collection of checkpoints during the training process as following:
| Model | Pre-trained | Mid-trained | Merged (i.e., WSM) |
|---|---|---|---|
| Ling-3.0-tiny | Ling-3.0-tiny-base-30T | Ling-3.0-tiny-base-midtrain | Ling-3.0-tiny-base |
| Ling-3.0-flash | Ling-3.0-flash-base-30T | Ling-3.0-flash-base-midtrain | Ling-3.0-flash-base |
These checkpoints correspond to different stages of the training process:
These checkpoints are released to support continued pretraining, fine-tuning, and further research. For the post-trained model, please see and see Ling-3.0-tiny and Ling-3.0-flash.
| Model Type | Base (final checkpoint of pre-training) |
|---|---|
| Architecture | Hybrid-linear MoE |
| Parameter Scale | Totoal 124B, Activated 5.1B (Non-emb) |
| Transformer Layers | 35 KDA + 7 Gated MLA (5:1) |
| Number of Dense Layers | 2 |
| Number of Routed Experts | 512 |
| Number of Shared Experts | 1 |
| Number of Activated Experts | 8 |
| Attention Heads | 32 |
| Hidden Size | 2560 |
| Expert Intermediate Size | 768 |
| Dense Intermediate Size | 6144 |
| Vocabulary Size | 157,184 |
To systematically assess the capabilities of the base model, we use a self-built comprehensive benchmark suite covering several key domains, including mathematics, coding, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e., Ling-3.0-flash-base, is compared below:
| Domain | Benchmark | Shot Config | Ling-3.0-flash-base | Ling-2.5-flash-base | NVIDIA-Nemotron-3-Super-base | Step-3.5-Flash-base | Hy3-preview-base | MiMo-V2.5-base |
|---|---|---|---|---|---|---|---|---|
| 124B A5.1B | 104B A7.4B | 120B A12B | 196B A11B | 295B A21B | 310B A15B | |||
| Knowledge | CCPM(EM) | 0-shot | 90.00 | 78.38 | 80.77 | 83.90 | 88.64 | 88.05 |
| ARC-C(EM) | 0-shot | 96.27 | 95.59 | 96.27 | 92.88 | 94.58 | 95.59 | |
| AGIEval(Acc) | 0-shot | 77.50 | 72.39 | 67.08 | 74.27 | 77.61 | 71.84 | |
| SimpleQA-Verified(Acc) | 5-shot | 26.90 | 15.90 | 21.20 | 27.80 | 24.20 | 19.10 | |
| MMLU-Pro (EM) | 5-shot | 67.74 | 61.36 | 63.89 | 63.65 | 66.44 | 64.90 | |
| CEval(EM) | 5-shot | 89.76 | 87.86 | 78.12 | 89.36 | 90.74 | 88.60 | |
| Code | HumanEval-Plus (Pass@1) | 0-shot | 81.10 | 80.49 | 78.05 | 78.05 | 79.27 | 76.22 |
| CruxEval(Pass@1) | 1-shot | 78.50 | 76.50 | 72.44 | 71.69 | 82.94 | 70.06 | |
| MultiPL-E(Pass@1) | 1-shot | 70.79 | 68.07 | 57.28 | 63.21 | 45.77 | 53.57 | |
| LiveCodeBench1 (Pass@1) | 1-shot | 40.09 | 33.04 | 38.99 | 35.68 | 35.90 | 35.50 | |
| BigCodeBench(Pass@1) | 0-shot | 52.19 | 50.35 | 25.53 | 49.74 | 52.02 | 50.18 | |
| FullStackBench(Pass@1) | 3-shot | 51.16 | 48.43 | 49.61 | 50.86 | 54.95 | 50.39 | |
| LCBench2 (Pass@1) | 3-shot | 57.76 | 57.51 | 54.22 | 49.67 | 46.94 | 45.12 | |
| Math | MATH500(Acc) | 4-shot | 79.00 | 74.00 | 70.80 | 62.80 | 60.60 | 70.00 |
| OlympiadBench (Acc) | 3-shot | 47.89 | 39.31 | 54.52 | 34.34 | 41.27 | 35.09 | |
| TheoremQA(Acc) | 5-shot | 61.10 | 57.95 | 72.05 | 56.03 | 60.96 | 51.23 | |
| OmniMath (Acc) | 3-shot | 47.65 | 36.20 | 38.55 | 28.46 | 33.33 | 27.03 | |
| Reasoning | CommonSenseQA (EM) | 5-shot | 89.93 | 87.55 | 86.24 | 88.62 | 86.98 | 84.60 |
| BBH (EM) | 3-shot | 89.17 | 84.72 | 89.00 | 87.19 | 76.00 | 86.22 | |
| Long-context | LongBench (Acc) | 0-shot | 52.62 | 42.81 | 31.05 | 21.47 | 20.58 | 29.57 |
| LEval (Acc) | 0-shot | 83.24 | 76.89 | 58.65 | 60.30 | 66.29 | 73.29 | |
|
Note: 1 LiveCodeBench (2408-2505) 2 LCBench (2301-2502) |
||||||||
Recommended use cases:
Not recommended as-is for:
For fine-tuning examples, please refer to our ling-cookbook.
If you have any question, please feel free to add a discussion.
This model is released under the MIT License.
model-00026-of-00027.safetensors 11 GBmodel-00025-of-00027.safetensors 10 GBmodel-00020-of-00027.safetensors 10 GBmodel-00003-of-00027.safetensors 10 GBmodel-00004-of-00027.safetensors 10 GBmodel-00006-of-00027.safetensors 10 GBmodel-00007-of-00027.safetensors 10 GBmodel-00008-of-00027.safetensors 10 GBmodel-00009-of-00027.safetensors 10 GBmodel-00010-of-00027.safetensors 10 GBmodel-00014-of-00027.safetensors 10 GBmodel-00016-of-00027.safetensors 10 GBmodel-00017-of-00027.safetensors 10 GBmodel-00021-of-00027.safetensors 10 GBmodel-00023-of-00027.safetensors 10 GBmodel-00018-of-00027.safetensors 10 GBmodel-00019-of-00027.safetensors 10 GBmodel-00001-of-00027.safetensors 10 GBmodel-00002-of-00027.safetensors 10 GBmodel-00012-of-00027.safetensors 10 GBmodel-00024-of-00027.safetensors 8.9 GBmodel-00013-of-00027.safetensors 8.1 GBmodel-00015-of-00027.safetensors 8.1 GBmodel-00022-of-00027.safetensors 8.1 GBmodel-00011-of-00027.safetensors 8.1 GBmodel-00005-of-00027.safetensors 8.1 GBmodel-00027-of-00027.safetensors 3.9 GBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo inclusionAI/Ling-3.0-flash-base-30T ./model-folder