chess-pgn-2400plus

A 51M-parameter character-level transformer that plays chess by writing PGN movetext, trained only on Lichess games where both players were rated 2400+. It follows the PGN model of Transcendence: Generative Models Can Outperform The Experts That Train Them (Zhang et al., NeurIPS 2024): it reads and writes games one character at a time (;1.e4 e5 2.Nf3 ...) and is never shown a board or the rules.

It is one of a pair trained with the same recipe on different rating bands: chess-pgn-1000-1300 and chess-pgn-2400plus.

Data

• Lichess standard rated games, March 2023 to June 2025 (28 months), newest first, both players rated 2400+, initial clock time of at least 180 seconds.
• 10,000,000 training games (4.82B characters, 482 per game on average) and 5,000 held-out games (the newest).
• Draws are kept. Games that are the source of a Lichess puzzle in our evaluation sets are excluded (22 dropped).
• Format: one game per line of text, ; + SAN movetext + result, e.g. ;1.e4 e5 2.Nf3 Nc6 ... 1-0. Vocabulary of 33 characters.

Model

Decoder-only transformer: 16 layers, d=512, 8 heads, ReLU MLPs, learned positions, context 1,024 characters (config.json).

Training

• Next-character cross-entropy on 1,024-character windows that each start at a game's beginning; 128 windows (131,072 characters) per step.
• AdamW (betas 0.9/0.95, weight decay 0.1), learning rate 2e-4 after 2,000 warmup steps, cosine decay to 2e-5 over 200,000 steps, bf16, one NVIDIA B200.
• Held-out loss every 500 steps; early stopping after 16 evals without improvement, counted after the first epoch.
• This model: the full 200,000-step schedule (its held-out loss was still improving at the last eval), then three fine-tuning stages, each starting from the previous stage's best weights with a fresh optimizer and ended by early stopping: peak learning rate 5e-5 (best after 500 steps), 2e-5 (best after 1,000) and 1e-5 after a 20,000-step warmup (best after 18,500). 220,000 optimizer steps in all on the path to this checkpoint (246,500 were run), about 6.0 passes over the training characters.

Results

Held-out loss: 0.3622 bits per character on the 5,000 held-out games.

Top move (greedy) equal to the move played, on 20,000 held-out positions (10,000 from each band's held-out games):

Illegal moves, on 3,000 held-out positions per band:

Usage

import json, chess, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import sys

path = snapshot_download("Arisp/chess-pgn-2400plus")
sys.path.insert(0, path)
from pgn_model import PGNTransformer, generate_moves

cfg = json.load(open(f"{path}/config.json"))["architecture"]
model = PGNTransformer(**cfg)
model.load_state_dict(load_file(f"{path}/model.safetensors"))
model.eval()

board = chess.Board()
board.push_san("e4")
move = generate_moves(model, [board], temperature=0.0, dev="cpu")[0]   # None if 5 attempts were illegal
print(board.san(move))

generate_moves writes characters until a space and parses them as SAN. At temperature 0 a retry repeats the same characters, so for play, sample (temperature > 0) on retries.

Limitations

• Illegal moves. The model only knows the position from the move history, and occasionally writes a move that is not legal.
• Context of 1,024 characters. Past roughly move 80-100 the start of the game no longer fits; the model then no longer knows the position and its move-match accuracy halves. Training never shows characters beyond a game's first 1,024.
• It imitates players of one rating band, blunders included; it is not an engine.