UDGAM LABS · Maths model · 95M parameters

Lilavati: Pocket Mathematician

A 95M-parameter English model, trained from scratch by Udgam Labs, that solves maths word problems by writing a short Python program and running it. It is small enough to run on a laptop CPU.

The name comes from Līlāvatī (लीलावती), the book of arithmetic word problems that the mathematician Bhāskara II wrote in 1150 CE.

More: try it in your browser · code and evaluation on GitHub · the checker model

Question: A chai stall sells 45 cups of tea in the morning and twice as many in the evening.
          Each cup costs 12 rupees. How many rupees does the stall earn in a day?

Lilavati writes:                                         Running it gives:
    morning_sales = 45                                   1620
    evening_sales = morning_sales * 2
    cup_price = 12
    total_sales = (morning_sales + evening_sales) * cup_price
    result = total_sales
    return result

Quick start

The lilavati package writes the programs, runs them in a restricted sandbox and combines the answers:

pip install torch transformers
git clone https://github.com/udgamlabs/lilavati && cd lilavati && pip install -e .

from lilavati import Lilavati
lv = Lilavati()      # downloads udgamlabs/Lilavati and udgamlabs/Lilavati-Checker

r = lv.solve("A shop sells pencils at 8 rupees each. Riya buys 6 pencils and 2 erasers at 5 rupees each. "
             "She pays with a 100 rupee note. How much change does she get?")
print(r["answer"])        # 42
print(r["agreement"])     # share of tries that gave this answer
print(r["program"])       # the program behind the answer

• lv.solve(q, tries=1) is one greedy try: fastest, no checker.
• lv.solve(q, tries=8) (default) writes 8 programs at temperature 0.7, runs them all, and picks the answer with the most checker-weighted support. More tries are slower but more accurate (see the evaluation).

Using the model directly. The prompt format is:

<bos>Question: {question}
Answer (Python):
```python

The model continues with a function and closes the code block. For several tries, sample at temperature 0.7 with top-k and top-p turned off, as in the evaluation. Requires transformers 5.18 or newer. Never run generated code without a sandbox. The lilavati package checks each program's structure (maths-only imports, no file or network access, no dunder access), then runs it in a separate process with time and memory limits, and under the macOS sandbox where available.

What it is good at, and what it is not

It is good at grade-school word problems: shopping and change, rates and speeds, percentages, discounts and tax, averages, simple interest, areas and perimeters, and multi-step arithmetic.

It is weak at:

• irrelevant information: a number that should be ignored often gets used anyway;
• working backwards and problems that add an extra step to a familiar pattern;
• unit conversions and some ratio questions;
• unanswerable questions: it always returns a number, even when the question has no answer;
• anything that is not a maths word problem. It is not a chatbot and has little general knowledge.

solve() returns agreement and confidence. When they are low (below 0.5), treat the answer as a guess.

Evaluation

All numbers were measured by us. "Code executed" means the model's program is run and its return value is compared with the reference answer.

One try (greedy), code executed:

Many tries with the checker (GSM8K test, 48 programs per problem at temperature 0.7):

With 16 tries, the checker-weighted vote scores 68.8.

Comparison with other open models up to 150M parameters

Measured by us on 14 and 15 October 2026, zero-shot and greedy, with one harness. Other models answer in text, their native format, and are scored on their boxed answer, or their last number if there is none. Output caps differ and are given in the notes (1K = 1,024, 8K = 8,192 and 16K = 16,384 new tokens).

Published results for models whose weights are not released: TinyGSM (Liu et al., 2023, arXiv:2312.09241) reports a 125M model, first pretrained on Phi-1.5 data, at 63.1% with one try and 68.9% when a 125M verifier picks one of 48 programs. Under that exact protocol (the checker's single top pick), Lilavati scores 66.7%. TinyGSM's 125M model trained from random initialisation reports 53.1%.

Contamination audit

We checked every pretraining document against each test set for shared 13-word sequences:

Removing the overlapping items barely changes the scores. With one greedy try, Lilavati scores, on the items with no overlap against the full sets: GSM-Plus 39.1 vs 39.2, SVAMP 69.6 vs 69.9 and ASDiv 75.3 vs 74.0. So this kind of overlap does not explain the results. A 13-gram check finds near-copies, not reworded problems. Problems added in later training were filtered against all these test sets (8-word sequences).

Where it falls short

All numbers are from the tables above.

• Perturbed and harder problems. One greedy try scores 54.7 on GSM8K test but 39.2 on GSM-Plus (the 7 answerable perturbation types) and 43.7 on GSM-Hard.
• Against TinyGSM. TinyGSM's 125M model, first pretrained on Phi-1.5 data, reports 63.1 with one try (Lilavati 54.7) and 68.9 when its verifier picks one of 48 programs (Lilavati with its checker, same protocol: 66.7). Those weights are not released.
• Choosing among tries. At least one of 48 programs is right on 89.5% of GSM8K problems, but the checker-weighted vote finds the right answer on 72.3%.
• The kinds of problem it gets wrong are listed under What it is good at, and what it is not.

How it works

1. Lilavati (this model) reads the question and writes a short Python function.
2. The program passes a structure check and runs in a restricted sandbox. Its return value is a candidate answer.
3. For more accuracy, Lilavati writes several programs. Lilavati-Checker, a second 95M model, reads the question and each program with its result and rates how likely it is to be right. The answer with the most checker-weighted support wins.

Training data:

• TinyGSM (MIT): about 12M grade-school problems with Python solutions;
• tinygsm_mixtral_12M: TinyGSM-style problems with solutions written by Mixtral;
• general English web and educational text;
• the GSM8K training split (MIT), and problems generated with large language models and verified by executing their solutions.

No test-set problems were used for training.

Intended use and limitations

Intended use: research and education on small reasoning models, program-aided maths, offline maths help on small devices, and as a starting point for fine-tuning.

Not suitable for: grading, financial or medical calculations, or anything where a wrong number could cause harm, without a person checking the program. A program that runs is not proof that it is correct.

The training problems are mostly in the style of English grade-school maths. Expect lower accuracy on other styles, longer problems, and other languages.

Licence

Weights and code: Apache-2.0.

Citation

@misc{lilavati2026,
  title  = {Lilavati: a 95M pocket mathematician that solves word problems by writing Python},
  author = {Udgam Labs},
  year   = {2026},
  url    = {https://huggingface.co/udgamlabs/Lilavati}
}

---

About Udgam Labs. Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. udgamlabs.com