KLM 1.6

KLM 1.6 is a 99M-parameter base language model that continues training from a checkpoint of our KLM 1.5 taken before its decay stage.

What's new

KLM 1.6 decays its learning rate over 100B tokens instead of 20B. Aside from that not much has changed, since this is an incremental update over KLM 1.5.

Benchmarks

All scores are zero-shot with length-normalized accuracy.

• ARC, HellaSwag and PIQA were scored using lm-eval 0.4.12.
• ArithMark-3 was scored using the official ArithMark-3.0 script.
• Peer scores are taken from the Open SLM leaderboard as of 2026-11-09.

On that leaderboard, KLM 1.6's Intelligence Index is 33.95 which puts it first among models under 100M, under 150M and under 250M parameters, and only two models of any size score higher, both of them around 360M parameters.

Each chart's axis starts at random-guess accuracy, which is 25% for most benchmarks and 50% for PIQA, so the height of a bar shows how far a model scores above chance.

Training

KLM 1.6 was trained on 2,048-token sequences in three stages, with 128 sequences per step in the first two and 192 in the decay.

1. Superposition warm-up for 70,000 steps, using Token-Superposition Training. Each input position is the average of 7 consecutive tokens, and the model learns to predict the next 7 tokens as an unordered group. This lets the warm-up cover 128B tokens of ClimbMix for the compute of about 18B.
2. Stable training on 80B tokens of next-token prediction at a constant learning rate.
3. Decay on 100B tokens while the learning rate falls to zero along a 1−√t curve.

The weight matrices were trained with MuonEq-R at a learning rate of 0.01, and the embeddings, norms and gates with AdamW at 3e-3, in bf16. Each 2,048-token sequence packs several documents, and the model never looks across a document boundary since attention is masked there, and the recurrent state and n-gram history are reset.

The training data came from the following sources.

Architecture

Intended use

KLM 1.6 is a base model. It continues text but has not been trained to follow instructions or chat, so it is meant to be fine-tuned for your own use case. The tokenizer already has <|im_start|>, <|im_end|>, <think> and </think> as single tokens, so they are ready to use in chat and reasoning fine-tunes.

Usage

The model code is plain PyTorch, so KLM 1.6 runs on a CPU as well as on any GPU without extra packages beyond Transformers.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MinimaLabs/KLM-1.6"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to(device)

inputs = tokenizer("The three primary colors are", return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• The model's knowledge is thin, and the facts it states are often wrong.
• It only works in English and has a 2,048-token context window.
• It has no safety alignment, so add your own filtering before putting it in front of users.

License

KLM 1.6 is released under the Apache-2.0 license, which you can find in LICENSE.