Kiyo-230M-Preview is an intermediate training checkpoint of Kiyo-230M, a decoder-only language model being pretrained from scratch on FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train. It is released as a preview: the training run is paused and this checkpoint has seen roughly one fifth of its planned token budget. See Preview Status below before using or comparing it.
The architecture is a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings, following the SmolLM2 architecture family and reusing its 49k-token vocabulary.
Preview Status
This repository holds the weights at optimizer step 81,500 of a run targeting 200B tokens. Three things follow from that, and all three matter when reading the benchmark numbers:
| Field | Value |
|---|---|
| Tokens seen by this checkpoint | 42,729,472,000 (42.7B) |
| Target token budget | 200,000,000,000 (200B) |
| Progress | 21.4% |
| Optimizer step | 81,500 |
| Planned total steps | ~381,470 |
| Learning rate at this checkpoint | 1.2e-3 (peak, stable phase) |
| LR decay begins at step | ~343,320 (final 10% of the run) |
The learning rate has not decayed yet. The run uses a warmup-stable-decay schedule whose decay phase covers only the final 10% of steps. This checkpoint sits deep in the stable phase, still at the peak LR of 1.2e-3. Models evaluated mid-plateau consistently score below what the same weights reach after annealing, so the numbers below understate the model rather than describe it.
It has seen fewer tokens than Kiyo-135M. Kiyo-135M was trained on the full 200B tokens. This preview has seen 42.7B. A parameter-count comparison between the two is not a fair comparison of the architectures — it is a comparison of a finished run against a fifth of one.
Weights are not final and will change. Do not treat this checkpoint as the Kiyo-230M release. It exists so the run can be inspected mid-flight; the final model will be published separately.
Model Details
| Field | Value |
|---|---|
| Parameters | 229,688,064 |
| Architecture | Llama-style decoder (SmolLM2 architecture family) |
| Layers | 32 |
| Hidden size | 768 |
| Intermediate size | 1,920 |
| Attention heads | 12 |
| KV heads | 4 |
| Head dimension | 64 |
| Attention type | Grouped query attention |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Positional encoding | RoPE (theta 10,000) |
| Vocabulary size | 49,152 |
| Context length | 2,048 |
| Embeddings | Tied input/output embeddings |
| Training tokens (this checkpoint) | 42,729,472,000 |
| Tokens per optimizer step | 524,288 |
| Peak learning rate | 1.2e-3 |
| LR schedule | Warmup-stable-decay (1/sqrt), decay not yet reached |
| Weight format | safetensors |
Training Data
| Source | Domain | Mixture share |
|---|---|---|
| FineWeb-Edu | General web text, education-filtered | 45.0% |
| DCLM-Baseline | General web text, high-quality filtered | 30.0% |
| Stack-v3-train | Source code | 17.5% |
| FineMath | Mathematical reasoning | 7.5% |
The Stack-v3 portion is sampled with per-language keep probabilities across 32 curated programming languages, giving each language an equal token quota rather than inheriting the corpus's natural skew.
Benchmarks
Scores are percentages from zero-shot acc_norm,none evaluations. Bold numbers indicate Kiyo-230M-Preview scores. Results may vary with evaluation settings.
| Benchmark | Kiyo-230M-Preview | Supra2-100M-Base | GPT-X2.5-135M | SmolLM2-135M |
|---|---|---|---|---|
| ARC-Easy | 49.54% | 47.81% | 51.81% | 58.63% |
| PIQA | 64.09% | 65.40% | 69.42% | 68.44% |
| ARC-Challenge | 26.11% | 24.83% | 29.18% | 29.69% |
| HellaSwag | 35.94% | 35.98% | 40.57% | 43.22% |
Usage
pip install -U transformers safetensors torch
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "DedeProGames/Kiyo-230M-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()
prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
output = model.generate(
input_ids=input_ids,
max_new_tokens=64,
do_sample=False,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations
This is an unfinished base model, not instruction-tuned — it continues text rather than following instructions, and its weights are an intermediate checkpoint rather than a release. Beyond the preview caveats above, it shares the limits of its scale: fluent and grammatical on language completion, competent on code, but weak on multi-step logic and context tracking, where it stays near chance on the harder items. The 2,048-token context window limits long-document use. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.
License
Apache 2.0