open-1b-midtrained-93B

open-1b is a 1.6 billion parameter decoder-only language model pretrained on 400B tokens. It is the first language model whose training can be independently verified.

Every open model release to date has asked that its users trust the account of how it was trained — open-1b allows you to check. It is released with its complete pretraining dataset, training and evaluation code, intermediate checkpoints at 100-step intervals, and a canonical state hash for every one of the 80,957 optimizer steps that produced it. Anyone can load the checkpoint before a given step, replay that step on their own hardware, hash the result, and confirm it matches the published fingerprint.

As organizations become more and more dependent on swarms of agents, we're reaching a point where AI verification is about to stop being a theory and start being a necessity. The only durable defense against models you cannot trust is models you can — models whose entire history is on the public record and can be replayed by anyone. That property cannot be added afterward; it has to be designed into the run from the first step. open-1b is our proof that this can be done in practice, and our attempt to set the standard for how.

This repository holds the midtrained checkpoint: the base model (Gensyn/open-1b-base) after continued pretraining on 93B additional tokens of the Dolma 3 Dolmino annealing mixture (allenai/dolma3_dolmino_mix-100B-1125, ingredient 1) — 23 sources spanning synthetic math, code, QA, reasoning and instruction data plus high-quality web and PDF text — with the learning rate annealed linearly to zero (OLMo 2-style midtraining recipe).

Model details

Model description

• Developed by: Gensyn
• Model type: decoder-only transformer, 1.61B total parameters (1.08B non-embedding)
• Language: English, with code and STEM text
• Licence: Apache 2.0
• Pretraining tokens: 400 billion, 80,957 steps
• Context length: 4,096 tokens
• Training cluster: 6 nodes, 48× NVIDIA H100
• Verification record: the record shows which training steps have accepted audits and which segments are confirmed

The open-1b family

Model sources

• Verification record: https://open1b.gensyn.ai
• Audit tool: https://github.com/gensyn-ai/pretraining-audit-cli
• Tech report: open1b.gensyn.ai/open1b-tech-report.pdf (arXiv soon)
• Pretraining data: gs://gensyn-open-1b/data — browse and search it at open1b.gensyn.ai/#/data/search
• Checkpoint trajectory (every 100 steps, full optimizer state): gs://gensyn-open-1b/ckpt
• Training code, configs and RepOps: https://github.com/gensyn-ai/open-transformers
• Per-step state hashes: served by the verification record

For what "auditable" means here — the per-step state hashes, the Merkle commitments, and how a replay lands on the same bytes — see the Gensyn/open-1b-base card.

Inference notes

The repo bundles its own modeling code (modeling_open1b.py), loaded with trust_remote_code=True — the architecture (gain-free QK-norm, embedding RMSNorm, block-aligned hybrid sliding-window attention) matches no stock transformers class. Parameter names and bytes in model.safetensors are identical to the training checkpoint the published state hashes commit to.

The model was trained with int8 W8A8 quantization-aware training (LSQ), and the learned per-channel weight_scale tensors ship in the checkpoint. By default the forward emulates the training int8 grid using those scales (config.quantized_forward=True); set it to False for plain bf16 GEMMs on the master weights. Inference logits are numerically close to, but not bit-identical with, the training stack (fp32 GEMM accumulation, bf16 attention vs the training int8 P·V flash kernel) — bit-exact replay of training steps is the job of the audit tool.

Load the tokenizer as-is. In particular, ignore transformers' suggestion to pass fix_mistral_regex=True: the "fix" changes the pre-tokenizer's behaviour, and tokenization bit-identical to training is part of this release's reproducibility contract.

Intended use

open-1b-midtrained is the recommended starting point for further training (e.g. supervised fine-tuning). Like the base model it is not tuned for chat or instruction following; use Gensyn/open-1b-sft for a chat-capable model.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Gensyn/open-1b-midtrained-93B", torch_dtype="bfloat16", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Gensyn/open-1b-midtrained-93B")