Model Details

This is a decoder-only base model with approximately 0.7B parameters following the Qwen3 dense architecture:

• Hidden Size: 1024
• Attention Heads: 16
• Layers: 28
• Sequence Length: 4096

Training Data

The model was trained from scratch on 100 billion tokens of the following data:

• English: HPLT 4.0 English Common Crawl subset

Tokenizer

The model uses the OpenEuroLLM tokenizer-256k with a vocabulary size of 262,144.

Training Information

The model was trained using NVidia-Megatron-LM (branch v0.15) on the LUMI HPC supercomputer. The training utilized 16 AMD MI250x nodes, totaling approximately 1500 GPU hours.

• Global batch size: 512 samples (approx. 2M tokens)
• Learning rate: 0.002 (WSD, 2000 iteration warmup, 10% iterations decay)

The full SLURM script used to run training is https://github.com/OpenEuroLLM/datamix-models-lumi/blob/main/train-0.4B-ne.sh. DATA_PATH and the WANDB_* settings were adjusted for the run.

Training logs

The model training logs are available on Weights & Biases: https://wandb.ai/openeurollm-project/datamix/runs/sj56j0ta https://wandb.ai/openeurollm-project/datamix/runs/ewqku3y1

Intermediate Checkpoints

We have released intermediate checkpoints to provide access to the model's training progression. These checkpoints are available in separate branches, with a new checkpoint released every 4000 training steps.

The naming convention is iter_0xxxxxxx. For example, the checkpoint for 16000 iterations is named iter_0016000. The available checkpoints range from iter_0004000 up to iter_0047684. The final checkpoint, iter_0047684, is located in the main branch.