Aurora2-400M-A20M is a decoder-only language model with approximately 400M total parameters and 19M active parameters per token. It uses a top-1 Mixture-of-Experts (MoE) architecture with 32 routed experts and one shared expert per layer.
The model was pretrained from scratch on 100B tokens using a single GPU. It is a base model, not an instruction-tuned assistant: there was no supervised instruction-tuning or RLHF stage.
| Property | Value |
|---|---|
| Total parameters | 399,597,454 |
| Active parameters per token | 19,050,382 |
| Experts | 32 routed + 1 shared per layer |
| Layers | 12 |
| Hidden size | 192 |
| Context length | 2,048 tokens |
| Vocabulary | 16,384 |
| Training tokens | 100B |
| Training precision | BF16 compute, FP32 master weights |
| Hardware | Single RTX Pro 6000, approximately 36 hours |
Aurora2 uses a relatively narrow attention stack paired with large, sparse feed-forward layers. Each token is routed to one of 32 experts, alongside the always-active shared expert.
| Component | Configuration |
|---|---|
| Hidden size | 192 |
| Layers | 12 |
| Attention heads | 6 query / 2 KV |
| Attention type | Grouped-query attention (3:1) |
| Head dimension | 32 |
| Positional encoding | RoPE, base 10,000 |
| Normalization | Pre-RMSNorm, FP32 QK normalization |
| Attention | FlexAttention, causal and per-document masking |
| Routed experts | 32 per layer, SwiGLU, FFN dimension 1,776 |
| Shared expert | 1 per layer, SwiGLU, FFN dimension 256 |
| Routing | FP32 softmax, top-1, dropless |
| Output head | Tied token embeddings |
| Logit soft-capping | 30 |
The router scales each selected expert's output by its routing probability. There is no expert capacity factor or token dropping.
Router training includes a Switch-style load-balancing loss (coefficient 0.01) and router z-loss (coefficient 0.001).
[1, 0].[0.5, 0.5].Aurora2 uses a byte-level BPE tokenizer trained from scratch.
"123" becomes three tokens).<|bos|>: ID 0<|eos|>: ID 1<|pad|>: ID 2<|bos|> to encoded text.During training, documents were packed together, with each document starting with a BOS token. No EOS was inserted between documents. BOS tokens also served as document boundaries for attention masking.
Training data was streamed from the Hugging Face Hub, tokenized on the fly and packed into sequences of 2,048 tokens. The tokenizer was trained on approximately 1B characters from the same data mixture.
Only training splits were used. ClimbMix shard 6542 was reserved for validation; benchmark evaluation splits were not used for training.
| Dataset | Purpose | Average token share | Approx. epochs |
|---|---|---|---|
| ClimbMix | General web text | 85.8% | < 1 |
| FineMath-4+ | Mathematical web text | 6.0% | < 1 |
| Wikipedia (English) | Encyclopedic text | 3.0% | < 1 |
| SmolTalk | Chat and QA | 2.5% | ~2.5 |
| OpenMathInstruct-2 | Mathematical QA | 1.5% | ~2 |
| NuminaMath-CoT | Mathematical QA | 1.2% | ~3 |
The mixture was adjusted throughout training. Math and QA/chat sources received greater weight during the final decay phase, while ClimbMix remained the majority source.
| Training stage | ClimbMix | FineMath | Wikipedia | SmolTalk | OpenMath | Numina |
|---|---|---|---|---|---|---|
| 0–60B tokens | 89.3% | 5.4% | 2.7% | 1.25% | 0.75% | 0.6% |
| 60–80B tokens | 85.8% | 6.0% | 3.0% | 2.5% | 1.5% | 1.2% |
| 80–100B tokens | 75.3% | 7.8% | 3.9% | 6.25% | 3.75% | 3.0% |
Instruction-style datasets were converted into plain-text formats, including Question: ... Answer: ..., Problem: ... Solution: ... and User: ... Assistant: .... These examples were part of pretraining; they do not constitute an instruction-tuning stage.
| Hyperparameter | Value |
|---|---|
| Total training tokens | 100B |
| Sequences per step | 128 |
| Sequence length | 2,048 |
| Tokens per step | 262,144 |
| Approximate optimizer steps | 381,000 |
| Learning-rate schedule | Warmup-Stable-Decay (WSD) |
| Warmup | 1B tokens, linear |
| Stable phase | 1B–80B tokens |
| Decay | Cosine decay over the final 20B tokens |
| Peak learning rate | 1.5e-3 |
| Router learning-rate multiplier | 0.5 |
| Optimizer | Muon + AdamW |
| Gradient clipping | 1.0 |
| Random seed | 1337 |
Muon was used for the attention projections and shared-expert matrices. It used momentum 0.95, five Newton-Schulz iterations and weight decay 0.1.
AdamW handled the token and n-gram embeddings, router parameters, normalization parameters, scalar/gating parameters and routed experts.
AdamW used betas (0.9, 0.95), epsilon 1e-8 and weight decay 0.1. Embeddings used weight decay 0.01, while norms, gates and scalar parameters had no weight decay.
Muon was deliberately not used on the routed experts because it negatively affected routing balance and training speed in experiments.
The following benchmark results were obtained for the 100B-token checkpoint.
Scores are shown as percentages. acc is standard accuracy, while acc_norm is normalized accuracy.
| Benchmark | acc | acc_norm |
|---|---|---|
| HellaSwag | 32.82 | 38.36 |
| ARC-Easy | 51.18 | 48.11 |
| ARC-Challenge | 22.95 | 26.11 |
| ARC | 37.07 | 37.11 |
| PIQA | 67.41 | 66.92 |
| ArithMark3 | 37.30 | 37.40 |
Intelligence Index (acc_norm): 21.52
The repository includes a native Transformers implementation with custom model code. You need trust_remote_code=True to load it.
Install the dependencies:
pip install -U transformers torch accelerate safetensors
Load the FP16 model:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AuroraAI-Research/Aurora2-400M-A20M"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo,
subfolder="fp16",
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
).eval()
# <|bos|> (id 0) is prepended automatically
inputs = tokenizer("The Capital of France is", return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.5,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The repository contains three precision variants:
| Directory | Weights |
|---|---|
| Root | FP32 |
bf16/ |
BF16 |
fp16/ |
FP16 |
The tokenizer is stored in the repository root, so it does not need a subfolder argument.
Hardware note: The inference implementation uses BF16 activations on CUDA, even when the stored weights are FP16. A CUDA GPU with BF16 support is therefore recommended.
Aurora2 is primarily intended for research into small language models, sparse Mixture-of-Experts architectures, routing strategies, optimization and training efficiency. It can also serve as a starting point for further fine-tuning.
A few limitations are worth keeping in mind:
@misc{aurora2_400m_a20m,
title = {Aurora2-400M-A20M: A Small Sparse Mixture-of-Experts Language Model},
author = {AuroraAI-Research},
year = {2026},
note = {Pretrained on 100B tokens on a single GPU}
}
model.safetensors 1.6 GBbf16/model.safetensors 799 MBfp16/model.safetensors 799 MB