Full-parameter continued pretraining of the base SmolLM2-135M model on the complete deduplicated BNC1994 training split. This is a next-token language model, rather than an instruction-tuned assistant.
Complete English notebook · Measured results · Run configuration · Experiment history
All targets in the held-out splits are evaluated using the same document-preserving 512-token protocol. NLL is summed over nonmasked next-token targets and divided by their total count; perplexity is exp(NLL). Validation selects the checkpoint; test does not.
| Split | Pretrained NLL | Pretrained PPL | Selected NLL | Selected PPL | PPL change |
|---|---|---|---|---|---|
| Validation | 3.252213 | 25.8475 | 3.112411 | 22.4752 | -13.05% |
| Test | 3.179929 | 24.0450 | 3.039538 | 20.8956 | -13.10% |
Validation covers 84 documents / 2,391,296 targets; test covers 77 documents / 2,669,143 targets. The test split has been evaluated in earlier stages and is not claimed to be untouched. These results demonstrate adaptation to this corpus distribution; they do not establish improved general reasoning or instruction following. Pretraining overlap with BNC is unknown.
93efa2f097d58c2a74874c7e644dbc9b0cee75a2; 134,515,008 parameters.Use the notebook with a legitimately obtained private corpus copy. All extraction, training, recovery, evaluation and publication source is included in one English notebook. The main executable path reproduces epoch 1; the full continuation/recovery executor in the appendix completes epochs 2–3 from its saved state. Historical attempts are clearly labelled and should not all be executed.
Development included a subset pilot, a full-data restart, a stopped 574-update continuation, three unsuccessful weight interpolations, and interrupted zero-update context preparations. Recovery replayed observed update positions:200,400 and 155 for the 512 branch; 350 for 1024; 242 UI-observed positions (225 timed positions) for 2048. Each recovery restored the same checkpoint weights, optimizer, scheduler, scaler and RNG. Causes of the unexpected runtime losses are unknown. Replay adds compute and is excluded from logical epoch counts. A complete cost/GPU-hour ledger is unavailable.
The independent 1024 and 2048 context comparisons each completed exactly two full epochs. The 1024 full held-out evaluation and export/reload verification passed. The 2048 checkpoint was frozen using complete validation results. Its complete held-out evaluation and export/reload verification also passed, and its T4 was released. Their best complete shared 512-token validation NLL/PPL were 3.113861 / 22.5078 and 3.115813 / 22.5518 respectively. Neither improved on the selected 512 model. Selection used the identical complete shared 512-token validation set, with a minimum replacement gain of 0.0005 NLL; test metrics did not determine selection. Supplementary evaluations at longer contexts are reported separately in context_comparison.json. Chunk boundaries, effective batch sizes and schedule details differ, so these runs are not a strict causal context-length ablation.
Development used multiple independent model initializations and concurrent single-T4 runtimes, exceeding the original exercise's one-model / overall-one-T4 restrictions. Each individual run used one T4. One selected model is published, and previous public revisions remain recoverable.
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "chao68681/smollm2-135m-bnc-continued-pretraining"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
inputs = tokenizer("The history of the British", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False,
pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))
This compact model can generate inaccurate, biased or memorized text. BNC reflects a historical British-English distribution, and aggregate perplexity does not guarantee useful responses. The corpus is not redistributed: no raw text, document/source identifiers, Arrow caches, optimizer states or private document-level diagnostics are included. The base model is released under Apache 2.0; see LICENSE. This model license grants no rights to the underlying BNC material; obtain and use that material under its applicable terms.
The exact published 512-token weights were identified by SHA-256 before evaluation. All 84 validation and 77 test documents were evaluated again to obtain document-level scores; the previously verified pinned-base document scores were reused. No optimizer ran. Aggregate NLL agrees with the existing complete evaluation within 1e-4. These are measurements of the published checkpoint, replacing the need to rely on historical epoch-one subgroup results.
| Split / kind | Documents | Base PPL | Published PPL | Relative PPL change |
|---|---|---|---|---|
| validation / all | 84 | 25.8475 | 22.4752 | -13.05% |
| validation / written | 72 | 25.6418 | 22.4418 | -12.48% |
| validation / spoken | 12 | 31.0472 | 23.2551 | -25.10% |
| test / all | 77 | 24.0450 | 20.8956 | -13.10% |
| test / written | 59 | 23.4949 | 20.8007 | -11.47% |
| test / spoken | 18 | 31.1272 | 21.9843 | -29.37% |
Paired source-group bootstrap: 2,000 replicates, seed 42. All six 95% intervals for selected-minus-base NLL are below zero. This is finite-corpus uncertainty, not evidence of out-of-domain capability. Full aggregate intervals and target counts are in subgroup_diagnostics.json; document identifiers and document-level records remain private. Test results did not select weights or hyperparameters.
Completed comparisons are recorded in context_comparison.json. The current model weights are unchanged; the prior verified public load remains applicable to the identical weight hash.