TinySQL-125M-WikiSQL

A 125M-parameter text-to-SQL model. Given a table schema and a question, it writes the SQL query. Every part of it was built from scratch: the tokenizer, the transformer, the optimizer, the training loop and the data pipeline. It was pretrained on 4.5B tokens on one GPU, then fine-tuned on WikiSQL.

• Paper: TinySQL: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens (Zenodo, 2026)
• Code: github.com/aditya0929/tinysql
• Base model: adiitya0929/TinySQL-125M-base
• feel free to contact me at - https://www.linkedin.com/in/aditya-narayan-jha-36193a251/

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "adiitya0929/TinySQL-125M-WikiSQL"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = (
    '<schema>CREATE TABLE t ("Player" TEXT, "No." TEXT, "Nationality" TEXT, "Position" TEXT, '
    '"Years in Toronto" TEXT, "School/Club Team" TEXT);</schema>\n'
    "<question>What position does the player who played for butler cc (ks) play?</question>\n"
    "<sql>"
)
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64)          # greedy; stops at </sql>
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
# SELECT "Position" FROM t WHERE "School/Club Team" = 'butler cc (ks)';

Prompt format. The model was trained on exactly this layout. The table is always called t. Column names are double-quoted, and each column is typed REAL (numeric) or TEXT. The question goes in <question>…</question>, and the prompt ends with <sql>. The model writes one SELECT statement and then </sql>.

Results (WikiSQL test, 15,878 questions)

The table reports execution accuracy (%): the generated query is run against the table and its result compared with the gold query's. Every model was fine-tuned with the same recipe after its own learning-rate sweep. This checkpoint is seed 0, which scores 78.01 greedy and 80.44 constrained. The official WikiSQL evaluator gives 78.08 execution and 72.38 logical-form accuracy for it.

model.generate() above is plain greedy decoding. The constrained numbers use the grammar-constrained decoder in the GitHub repository (decode/). It only allows the table's real column names and values copied from the question, which gives 100% valid SQL.

Training

The architecture matches Llama, so the weights are distributed in the standard LlamaForCausalLM format. The model was built and trained with the repository's own code, not with transformers. The export is checked to produce the same token ids, logits and generated SQL as the original implementation.

Other checkpoints

The weights of seeds 1 and 2 and of the other baselines' test runs were not kept.

Limitations

• WikiSQL shape only: one table, one selected column, an optional aggregate, and up to four AND conditions. No joins, GROUP BY, ORDER BY or nesting. Do not expect it to handle Spider-style or production SQL.
• English only; small model; greedy decoding can produce column names that don't exist. Use the constrained decoder from the repository to rule that out.
• Label noise: WikiSQL's labels are noisy. A hand audit in the paper suggests that most remaining aggregation errors are label errors.
• Not for production databases without review: generated SQL should never be executed with write permissions.

Training data licenses

FineWeb-Edu (ODC-By 1.0), The Stack (dedup), SQL subset (permissively licensed source files, subject to The Stack's terms of use), Stack Exchange dba and datascience (CC BY-SA; 2.5, 3.0 or 4.0 depending on the post date), gretelai/synthetic_text_to_sql (Apache-2.0), WikiSQL (Salesforce, BSD-3-Clause). The pretraining data was decontaminated against WikiSQL dev and test questions.

Citation

@techreport{jha2026tinysql,
  title       = {{TinySQL}: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens},
  author      = {Jha, Aditya Narayan},
  year        = {2026},
  institution = {Zenodo},
  doi         = {10.5281/zenodo.23396770},
  url         = {https://doi.org/10.5281/zenodo.23396770}
}