Haidass-143M
English | 中文
A bilingual (English/Chinese) small language model trained entirely on Huawei Ascend NPU ecosystem.
Model Overview
Haidass-143M is a 143M-parameter bilingual language model trained on approximately 100B tokens of English and Chinese data. The entire training pipeline runs on the Huawei Ascend ecosystem, using the MindSpeed-LLM framework on Atlas A2 servers (910B). A custom 64,000-token bilingual vocabulary (SentencePiece BPE) was trained alongside the model. This model is competitive among multilingual models under 150M parameters and ranks favorably across multiple evaluation benchmarks.
Model Architecture
| Parameter | Value |
|---|---|
| Architecture | Qwen3 |
| Layers | 30 |
| Hidden size | 576 |
| Attention heads | 9 |
| KV heads (GQA) | 3 |
| Head dim | 64 |
| FFN intermediate size | 1,536 |
| Vocabulary size | 64,000 |
| Max sequence length | 4,096 |
| Tie word embeddings | Yes |
| Position encoding | RoPE (θ=100,000) |
| Attention bias | None |
| Precision | BF16 |
| Total parameters | ~143M |
Training Data
The model was trained on approximately 100B tokens of mixed English and Chinese data. Primary data sources:
- openbmb/Ultra-FineWeb (ultrafineweb-en + ultrafineweb-zh)
- mlfoundations/dclm-baseline-1.0-parquet (dclm)
- HuggingFaceTB/finemath (finemath-4plus)
Training Configuration
| Parameter | Value |
|---|---|
| Framework | MindSpeed-LLM (v2.3.0) |
| Hardware | 8 × Atlas A2 servers (8 NPUs per node, 256 cores) |
| NPU model | Huawei Ascend 910B |
| Total NPUs | 64 (8 nodes × 8 cards) |
| Sequence length | 4,096 |
Optimizer
| Parameter | Value |
|---|---|
| Optimizer | AdamW |
| Peak learning rate | 3e-4 |
| Min learning rate | 3e-5 |
Tokenizer
| Property | Value |
|---|---|
| Type | SentencePiece BPE |
| Vocabulary size | 64,000 |
| Language coverage | English + Chinese |
Evaluation
Evaluated at checkpoint (~98B tokens) using the lighteval framework (v0.9.2).
| Benchmark | Score |
|---|---|
| ARC-Easy | 60.44 |
| ARC-Challenge | 27.13 |
| PIQA | 67.25 |
| HellaSwag | 37.91 |
| OpenBookQA | 31.8 |
| Winogrande | 52.17 |
| agi_eval | 23.78 |
Key Features
- Fully Ascend-native: Trained entirely on Huawei Ascend 910B NPUs using the MindSpeed-LLM framework
- Bilingual: Trained on a mixture of English and Chinese data
Intended Use
This is a research model, suitable for:
- Studying training dynamics of small models on Ascend NPUs
- English/Chinese language modeling research
- Serving as a base model for fine-tuning or annealing experiments
Limitations
- Small model scale; reasoning and generation capabilities are limited
- raw pretrained model only
Citation
@misc{haidass-143m,
title={haidass-143M: A Bilingual Small Language Model Trained on Ascend 910B},
year={2026},
note={Based on Qwen3 architecture, trained from scratch on 100B tokens using MindSpeed-LLM on 64× Ascend 910B NPUs}
}
License
Apache 2.0