UDGAM LABS · Checker model · 95M parameters

Lilavati-Checker

A 95M-parameter model that reads a maths word problem, a Python program written by Lilavati and the program's result, and estimates how likely the result is to be right. Lilavati uses it to choose between many tries.

More: Lilavati · try it in your browser · code and evaluation on GitHub

How it is used

Lilavati writes several programs for a question and runs each one. The checker rates every program. Each candidate answer gets the sum of the ratings of the programs that produced it, and the answer with the most support wins. This is what Lilavati.solve() does by default:

from lilavati import Lilavati
lv = Lilavati()                                  # loads Lilavati and Lilavati-Checker
r = lv.solve("Meera deposits 25,000 rupees at 8% simple interest per year. How much interest does she earn in 3 years?", tries=8)
print(r["answer"], r["confidence"])              # 6000, and the checker-weighted share of support for it
print(r["scores"])                               # the checker's rating of each of the 8 programs

Using the model directly. Build this text (with <bos> first), run the model, and compare the next-token logits for " yes" and " no" at the last position. P(correct) is the softmax of those two logits.

<bos>Question: {question}
Answer (Python):
```python
{program}
```
Result: {result, or "error" if the program failed}
Correct:

Keep the last 768 tokens if the text is longer.

Evaluation

GSM8K test (1,319 problems). Lilavati wrote 48 programs per problem at temperature 0.7, and each program was run.

Lilavati alone, one greedy try: 54.7.

For comparison, TinyGSM (Liu et al., 2023, arXiv:2312.09241) reports 68.9% when its 125M verifier picks one of 48 programs from its 125M generator. That generator was first pretrained on Phi-1.5 data, and neither model's weights were released. Under that same protocol (the checker's single top pick), Lilavati with this checker scores 66.7%.

Where it falls short

• It was trained only on training-set problems (no test-set problems) and only on programs in Lilavati's style. Its ratings of programs from other models are untested.
• It can be confidently wrong, especially when most tries share the same mistake, such as using a number that should have been ignored.
• It judges grade-school maths word problems. It is not a general code or proof verifier.
• A high rating is not a guarantee. Check answers that matter.

Licence

Weights: Apache-2.0.

Citation

@misc{lilavati2026,
  title  = {Lilavati: a 95M pocket mathematician that solves word problems by writing Python},
  author = {Udgam Labs},
  year   = {2026},
  url    = {https://huggingface.co/udgamlabs/Lilavati}
}

---

About Udgam Labs. Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. udgamlabs.com