İvme-Conversate-v1-Base (Codenamed Apple 1)
İvme (Turkish: acceleration) is a series of stupidly small language models built to punch above their weight. This is the first release: a 22M parameter decoder-only base model trained from scratch on a dense, quality-filtered corpus.
The goal is not production deployment. The goal is to see how well a sub-25M model can perform when every decision — architecture, data mix, optimizer, training schedule — is made deliberately.
Model Details
| Parameter | Value |
|---|---|
| Architecture | Decoder-only transformer |
| Parameters | 22,028,160 |
| Layers | 10 |
| Hidden dim | 384 |
| FFN dim | 1024 (SwiGLU) |
| Attention heads | 6 query / 2 KV (GQA) |
| Context length | 1024 tokens |
| Vocab size | 16,384 (custom BPE) |
| Positional encoding | RoPE (θ=10,000) |
| Normalization | RMSNorm (pre-norm) |
| Embeddings | Tied input/output |
| Biases | None |
Benchmarks
All benchmarks run via EleutherAI lm-evaluation-harness, 0-shot. WikiText-2 uses byte_perplexity for tokenizer-independent comparison.
| Benchmark | Score | Notes |
|---|---|---|
| WikiText-2 (byte_perplexity) ↓ | 2.96 | Lower is better |
| BLiMP ↑ | 61.40% | Average over 67 subtasks; random baseline 50% |
| ARC-Easy ↑ | 30.85% | acc_norm, 0-shot |
Training
Data Mix (~1.57B tokens, Chinchilla-optimal)
Data is ordered in ascending quality for curriculum learning — the model sees noisier web text first and the densest material last.
| Source | Tokens | Share |
|---|---|---|
| epfml/FineWeb-HQ (score > 0.8) | ~710M | 45% |
| bigcode/python-stack-v1-functions-filtered | ~160M | 10% |
| HuggingFaceTB/finemath (finemath-4plus) | ~235M | 15% |
| HuggingFaceTB/cosmopedia (stanford + wikihow) | ~395M | 25% |
| wikimedia/wikipedia (EN, 20231101) | ~80M | 5% |
Hyperparameters
| Setting | Value |
|---|---|
| Optimizer | Muon (body weights) + AdamW (embeddings, norms) |
| Muon lr | 0.02 |
| AdamW lr | 3e-4 |
| LR schedule | Warmup-Stable-Decay (WSD) |
| Warmup steps | 100 |
| Decay fraction | 20% of training |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Effective batch | ~1.05M tokens/step |
| Total steps | 1,447 |
| Precision | bfloat16 |
| Attention | Flash Attention 2 (HF Kernels) |
| Final weights | EMA (β=0.999) of training trajectory |
Hardware
Trained on a single NVIDIA RTX PRO 6000 Blackwell (96GB) in approximately 20 minutes.
Tokenizer
Custom BPE tokenizer trained from scratch on a balanced sample of the pretraining corpus. Vocab size 16,384 with ByteLevel pre-tokenization.
Special tokens: <|pad|>, <|bos|>, <|eos|>, <|unk|>, <|user|>, <|assistant|>, <|system|>
Usage
import torch
from tokenizers import Tokenizer
# Load with custom code (not a standard HF AutoModel — see model.py)
from model import IvmeConfig, IvmeConversate
tokenizer = Tokenizer.from_file("ivme_tokenizer.json")
ckpt = torch.load("ivme_base_ema.pt", map_location="cuda", weights_only=False)
cfg = ckpt["cfg"]
cfg.attn_backend = "sdpa" # or "kernels" for HF Kernels flash-attn
model = IvmeConversate(cfg).cuda()
model.load_state_dict(ckpt["model"])
model.eval()
prompt = "The theory of relativity states that"
ids = torch.tensor([tokenizer.encode(prompt).ids], device="cuda")
out = model.generate(ids, max_new_tokens=100, temperature=0.8, top_k=40)
print(tokenizer.decode(out[0].tolist()))
Limitations
- Base model only — not instruction tuned, will not follow instructions or answer questions
- English only (v1)
- Limited factual knowledge due to Chinchilla-optimal training (1.57B tokens)
- Repetition at higher temperatures without
repetition_penalty - 1024 token context window
What's Next
- İvme-Conversate-v1-Instruct — Sadly cancelled after not performing as expected with tuning.
- İvme-Conversate-v2 — extended training (~13B tokens), new model architecture.
You can check our other upcoming models on our organization card!
Citation
@misc{ivme-conversate-22m,
author = {IvmeLabs},
title = {İvme-Conversate-v1-Base},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/IvmeLabs/Ivme-Conversate-22M-Base}
}
Credits
The apple photo by Matheus Cenali on Unsplash
Built by IvmeLabs. Small models, deliberate choices.