Qwen2.5-7B Bookworm

How far can you push a 7B model to learn a whole book, without breaking what it already knows?

Bookworm is Qwen2.5-7B-Instruct taught Russian non-fiction books one after another, a fresh LoRA per book on the previous book's merged weights. Book 1 is a complete book (2,176 passages); books 2 and 3 are Russian Wikipedia on Soviet computers (409 passages) and Soviet home computers (431 passages), close topics on purpose. After all three the model answers questions from every book, rejects false claims about them, keeps 87% of what it knew of book 1 and 89% of book 2, and still scores above the base model on IFEval and GSM8K and within 0.7 of it on MMLU.

The weights are private for now (book 1's text is not mine to redistribute, and a model trained to recite a book can recite it). This card documents the method and the measurements.

Results

Forgetting, counted per question (of the questions a model got right, how many are still right later):

• Book 1, right after book 1 (91): 85 still right after book 2 (0.93), 79 after book 3 (0.87). The totals hide some churn: a few questions flip each way, so book 1's total moves less than its per-question retention.
• Book 2, right after book 2 (377): 337 still right after book 3 (0.89), with book 3 on a neighbouring topic.
• Each new book: book 2 from 19 to 377 of 422 (89%), book 3 from 6 to 435 of 481 (90%).
• General skills after three books: the only drop below base is MMLU, −0.70.

Base Qwen scores 28% on the book-2 exam because Wikipedia is partly familiar; after book 1 the model answers in the books' short, sourced style and stops guessing (19 and 6 before training), so each book is learned rather than recalled. Base answers also average ~1,300 characters against ~150, which flatters the base model on a key-fact metric.

All numbers: one user turn, no system prompt, greedy decoding, merged fp16 weights, lm-evaluation-harness 0.4.13 on vLLM. Each book exam was written independently of the training rows (questions too close to a training question were dropped), and an answer counts as correct when it contains at least 60% of the question's key facts. Book 1's exam covers its first 140 passages.

Training time: book 1 took 3.7 hours end to end on one A100 80 GB, data generation included; books 2 and 3 took 52 minutes each on one H100 (about 30 min of data generation with the open-weight teacher on the same card, 20–23 min of training).

Why Russian

Russian was the perfect language to stress-test this.

• The model has the weakest prior here. Qwen2.5-7B is strongest in English and Chinese. In Russian it starts from less fluency and almost no knowledge of the book, so every fact has to actually be learned, not recalled.
• Cyrillic costs more tokens per fact, so the same book is a bigger training load than it would be in English.
• The benchmarks are in English. Pouring ~120,000 Russian training rows into a model is the fastest way to make it drift: it starts answering in Russian, opening with «Нет», or formatting maths answers the way it formats book answers. Any of that shows up immediately in IFEval, GSM8K and MMLU. If general ability holds here, it holds.

How it was trained

Data (all generated with open-weight models, no closed-model text anywhere):

• An open-weight teacher (Gemma 4 26B-A4B on vLLM) writes questions about every passage in several forms (direct, why, what-if-not, situational) plus rephrasings; a judge keeps a question only if its passage really answers it.
• Answers are the passage itself, word for word. The model recites well and invents when it paraphrases, so the training target is the source text, not a teacher's summary.
• Traps: false-premise, swapped-attribute and spliced questions built from each passage and its nearest neighbours, answered «Нет, это не так.» followed by the true passage. Each trap has a true twin question, so the model learns to check the claim instead of refusing everything that looks like a trap. Near-miss traps (one swapped slot, a splice of two neighbouring passages, a false presupposition) halved made-up answers.
• General rows: the base model's own answers to everyday, formatting and maths prompts, plus plain-text 5-shot maths completions, to keep its behaviour outside the book.
• Contamination guard: no training row may share an 8-gram with any IFEval, GSM8K or MMLU test item.

Training (one objective, two kinds of rows):

• Cross-entropy on the book rows; KL to the starting model (cached top-64 logits) on the general rows, so the update is pulled back wherever it is not about the book.
• LoRA r64 / α128 on all linear layers, 2 epochs, lr 5e-5 cosine, gradient clipping 1.0, packed sequences, Unsloth.

What it took to get here

The interesting part is what failed on the way. Each line is a measured run on the whole book or a 140-passage slice:

A few lessons that held across every run:

• Damage to general ability grows with the amount of book: without the KL anchor, IFEval fell 4 points at 140 passages, 18 at 500 and 36 on the whole book. Test at full size.
• Read GSM8K both strict and flexible: most "maths damage" was a lost answer format, not lost reasoning.
• One epoch instead of two costs ~7 knowledge points and weakens the trap reflex.

How the later books are protected

Each later book trains a fresh LoRA on the previous book's merged weights, with KL to that model on general and plain-text rows, and replays 3 rows per passage of every earlier book (1 question, 1 trap, 1 twin) inside the same update. No extra modules, no adapter switching, no retrieval: one checkpoint answers questions from all three books.

Author

Mihir Karbelkar, software engineer with ML experience. Open to fine-tuning work: knowledge injection, continual learning, evaluation. Hugging Face · GitHub