open-1b-sft

open-1b is a 1.6 billion parameter decoder-only language model pretrained on 400B tokens. It is the first language model whose training can be independently verified.

Every open model release to date has asked that its users trust the account of how it was trained — open-1b allows you to check. It is released with its complete pretraining dataset, training and evaluation code, intermediate checkpoints at 100-step intervals, and a canonical state hash for every one of the 80,957 optimizer steps that produced it. Anyone can load the checkpoint before a given step, replay that step on their own hardware, hash the result, and confirm it matches the published fingerprint.

As organizations become more and more dependent on swarms of agents, we're reaching a point where AI verification is about to stop being a theory and start being a necessity. The only durable defense against models you cannot trust is models you can — models whose entire history is on the public record and can be replayed by anyone. That property cannot be added afterward; it has to be designed into the run from the first step. open-1b is our proof that this can be done in practice, and our attempt to set the standard for how.

This repository holds the supervised fine-tune (SFT): the midtrained model (Gensyn/open-1b-midtrained-93B) fine-tuned on allenai/tulu-3-sft-olmo-2-mixture-0225. This is an SFT-only model — no RLHF, DPO or other preference tuning was applied. It uses the chat template shipped in tokenizer_config.json (apply it with tokenizer.apply_chat_template).

Model details

Model description

• Developed by: Gensyn
• Model type: decoder-only transformer, 1.61B total parameters (1.08B non-embedding)
• Language: English, with code and STEM text
• Licence: Apache 2.0
• Pretraining tokens: 400 billion, 80,957 steps
• Context length: 4,096 tokens
• Training cluster: 6 nodes, 48× NVIDIA H100
• Verification record: the record shows which training steps have accepted audits and which segments are confirmed

The open-1b family

Model sources

• Verification record: https://open1b.gensyn.ai
• Audit tool: https://github.com/gensyn-ai/pretraining-audit-cli
• Tech report: open1b.gensyn.ai/open1b-tech-report.pdf (arXiv soon)
• Pretraining data: gs://gensyn-open-1b/data — browse and search it at open1b.gensyn.ai/#/data/search
• Checkpoint trajectory (every 100 steps, full optimizer state): gs://gensyn-open-1b/ckpt
• Training code, configs and RepOps: https://github.com/gensyn-ai/open-transformers
• Per-step state hashes: served by the verification record

For what "auditable" means here — the per-step state hashes, the Merkle commitments, and how a replay lands on the same bytes — see the Gensyn/open-1b-base card.

Inference notes

The repo bundles its own modeling code (modeling_open1b.py), loaded with trust_remote_code=True — the architecture (gain-free QK-norm, embedding RMSNorm, block-aligned hybrid sliding-window attention) matches no stock transformers class. Parameter names and bytes in model.safetensors are identical to the training checkpoint the published state hashes commit to.

The model was trained with int8 W8A8 quantization-aware training (LSQ), and the learned per-channel weight_scale tensors ship in the checkpoint. By default the forward emulates the training int8 grid using those scales (config.quantized_forward=True); set it to False for plain bf16 GEMMs on the master weights. Inference logits are numerically close to, but not bit-identical with, the training stack (fp32 GEMM accumulation, bf16 attention vs the training int8 P·V flash kernel) — bit-exact replay of training steps is the job of the audit tool.

Load the tokenizer as-is. In particular, ignore transformers' suggestion to pass fix_mistral_regex=True: the "fix" changes the pre-tokenizer's behaviour, and tokenization bit-identical to training is part of this release's reproducibility contract.

Chat format

The chat template (shipped in tokenizer_config.json and chat_template.jinja) renders every turn as

<|start_header|>{{role}}<|end_header|>

{{content}}<|eot|>

using the tokenizer's reserved special tokens (<|start_header|> id 2, <|end_header|> id 3, <|eot|> id 4 — all atomic single-id encodes). There is no sequence-level BOS prefix. <|eot|> terminates every turn and is the model's eos for generation, so generate() stops at the end of the assistant turn.

The generation prompt deliberately ends at <|end_header|> (no trailing newlines) and the model generates the \n\n itself — that transition follows every header in its training data. Completions therefore start with two newline characters; strip leading whitespace from the decoded text. Do not append the two newlines to the prompt yourself: encoded at the end of a prompt they merge into a single token the model never saw at that position (assistant content always followed in training, splitting them), and the model responds to it with an immediate <|eot|> — an empty generation. If you must reproduce the exact training-style prompt (e.g. for scoring), append a sentinel character before encoding and strip its tokens afterwards.

Intended use

open-1b-sft is the chat-capable member of the family, intended for research use and as a demonstration that the verifiable base model fine-tunes cleanly. It has had supervised fine-tuning only: expect a helpful but lightly-tuned assistant, not a production chat model. Always prompt it through the chat template.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Gensyn/open-1b-sft", torch_dtype="bfloat16", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Gensyn/open-1b-sft")

messages = [{"role": "user", "content": "Explain verifiable training in one paragraph."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())