classical-smollm2-trunk: a language model that has only read pre-1868 texts

A 125M-parameter language model trained from scratch on about 1 billion tokens of texts written before 1868: classical Japanese, Edo-period Japanese and Literary Chinese. It has never seen modern Japanese, English or anything from the web.

It is a research artifact for studying what a model learns from premodern text alone: its vocabulary, its "world knowledge" and its style. It also gives a period-faithful baseline for working with historical documents. It is a base model: it continues text and does not follow instructions. For a conversational version, see classical-smollm2-sft.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "kuramot6f/classical-smollm2-trunk"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16).eval()

inputs = tok("春はあけぼの", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, top_p=0.95, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is needed only for the small SentencePiece tokenizer wrapper (classical_tokenizer.py). The model is a standard LlamaForCausalLM.

Model

Training

• Data: classical-ja-pretrain, train split.
• 30,518 updates, bf16, AdamW (weight decay 0.1), warmup-stable-decay learning-rate schedule. Single GPU, PyTorch + 🤗 Transformers + Accelerate.
• Data mixture: most of training sampled about 50% NDL, 48.5% Kanripo, 1.25% Wikisource and 0.25% Minna de Honkoku. During the final learning-rate decay (updates 24,415–30,518), the mixture moved toward the curated literary sources: 65% Wikisource, 15% Minna de Honkoku, 10% NDL, 10% Kanripo.
• This repo holds the final checkpoint. training_config.json has the full configuration and metrics.jsonl the per-step loss and learning rate.

Limitations

• It is small: generations drift and repeat after a few sentences.
• About 44% of the training tokens are uncorrected OCR, so the model sometimes reproduces OCR noise.
• It knows only what pre-1868 texts say, and their views, including religious, political and social views that may be offensive today. It must not be used as a source of facts.

License

Weights and code: Apache-2.0.

Training data note: the model was trained on texts under several licenses. These are public domain (NDL OCR); CC BY-SA 4.0 (Kanripo, Wikisource, Minna de Honkoku); and CC BY-NC-SA 3.0 TW for CBETA-derived Buddhist texts, about 13% of the corpus tokens. See the dataset LICENSES. The license on the weights does not change the licenses of the training texts. If the model reproduces a passage from its training data, that passage stays under its original license.