GoldWorm v6 — Byte-Level Causal Transformer (2.76M params)

Try this model

Open Gold Lab Space

Verified result: full BPC 1.007349 (step 8000, TinyStories-valid, 129,429 fixed outer-val positions). Beats GPT-2 124M BYTE_BPC = 1.142985 (TOKEN_BPC = 9.152823) on the same metric.

Dataset: roneneldan/TinyStories (TinyStories-valid.txt, 19.4 MB, sha256 94e431816c4cce81...). Protocol: windowed stride-16 ctx-256 over fixed evaluation set; determinism verified (two bit-identical passes, max_abs_diff_per_position_bits = 0.0).

Model facts

• Type: ByteTransformer
• Vocab: 256 (byte-level)
• Config: vocab_size=256, d_model=192, n_layer=6, n_head=4, ffn_mult=4, ctx=256, tie_embeddings=True, dropout=0.0
• Params: 2,757,504
• Checkpoint: run/andrew_8000/checkpoint_selected.pt → model.safetensors
• Selection (inner-val): step 8000, inner_val_windowed_bpc = 1.051187
• Full-val (outer): step 8000, full_bpc = 1.007349
• Baselines: unigram 4.437 / bigram 3.317 (add-k 0.01, train-matched)
• Gate: well below frozen v1 reference (2.566)

Files

• model.safetensors — exported weights (tied embeddings cloned; load with ByteTransformer(config))
• config.json — architecture config
• eval_results.json — full outer-val result
• demo/app.py — Streamlit text-generation demo
• demo/requirements.txt — NOT in repo (dropped); install torch safetensors streamlit numpy manually

Use with Streamlit demo

pip install -r demo/requirements.txt
streamlit run demo/app.py

Prompt is drawn from corpus/corpus.txt outer-val region; generation uses temperature / top-k sampling.

How to load (Python)

import torch
from safetensors.torch import load_file

cfg = TransformerConfig(vocab_size=256, d_model=192, n_layer=6, n_head=4, ffn_mult=4, ctx=256, tie_embeddings=True, dropout=0.0)
model = ByteTransformer(cfg)
state = load_file("model.safetensors")
model.load_state_dict(state)
model.eval()

Verification

• Chunk shape / parity / replication checks verified per LLM_PARITY_ROADMAP.md
• Independent replication (step 8000) confirmed bit-exact with selection frozen before any outer_val read

Version label

GoldWorm v6, 2.76M params, single release

User manual

See docs/USER_MANUAL.md for Windows + Linux install + load + eval reproduction

See also: Uuuuuuniiiiiii/TinyStories dataset card