TinySQL: LoRA Fine-Tuning of a Transformer Built from Scratch
A custom, 439,552-parameter PyTorch SQL Transformer trained from random initialization, with a matching 32,768-parameter rank-8 LoRA adapter for synthetic English question-to-SQL generation.
Interactive base-vs-LoRA demo · GitHub source · Dataset card · Experiment report
Checkpoint identity matters: This Hugging Face repository contains a freshly trained base/adapter pair, not the separately stored historical
base_small/aug_r8_fastcheckpoints. The results below come from the three publishedevaluation-*.jsonfiles for these uploaded weights. The adapter is custom PyTorch LoRA code, not a standard Hugging Face PEFT adapter.
At a glance
| Property | Published model and adapter |
|---|---|
| Task | English question → SQLite SQL, constrained to a fixed synthetic schema |
| Base architecture | 2-layer decoder-only causal Transformer, width 128, four heads |
| Base model | 439,552 parameters, originally trained from random initialization |
| LoRA adapter | 32,768 trainable parameters, rank 8, alpha 16 |
| Base + LoRA | 472,320 parameters, 6.94% trainable during LoRA phase |
| Tokenizer | Custom task-specific tokenizer; vocabulary 174 |
| Context | 160 tokens |
| Training data | Generated synthetic e-commerce questions and SQL pairs |
| LoRA test execution | 89.06% on 320 synthetic examples |
| LoRA challenge execution | 21.88% on 160 harder synthetic examples |
| Base compatibility | Exact paired base/model.pt and base/tokenizer.json required |
| Implementation | Custom PyTorch and custom LoRA; not AutoModel / PEFT |
| License | MIT |
Model details
Model description
TinySQL is a small causal language model for generating SQL continuations from natural-language questions and a fixed database schema. The base model was independently trained from scratch using supervised question/SQL pairs; a subsequent low-rank adaptation (LoRA) phase freezes the base parameters and optimizes only low-rank updates to selected projections.
The fine-tuning data contain extra synthetic paraphrases of the original training set. The LoRA adapter changes the existing model's behavior but does not turn it into a general-purpose SQL assistant.
Developed and shared by: Nehal Vaghasiya
Model type: Decoder-only causal Transformer with custom low-rank adapters
Language: English questions, generated SQLite SQL
Base model: Custom TinySQL base trained from random initialization; included in this repository under base/
License: MIT
Source commit: 5fdb74437670658c928e1f0fa4dacf80fca9c0e1
Fixed schema
The generated questions and queries assume only the following fabricated tables:
customers(id, name, state)
orders(id, customer_id, total, status)
products(id, name, category, price)
The demonstration executes generated SQL only against an isolated synthetic SQLite database, not a real business system.
Published files and provenance
| Path | Purpose |
|---|---|
base/model.pt |
Exact matched frozen base PyTorch weights |
base/tokenizer.json |
Matching custom token vocabulary |
base/metadata.json |
Base training configuration, provenance and best validation loss |
base/history.json |
Base training history |
adapter.pt |
Custom rank-8 LoRA adapter weights |
metadata.json |
LoRA settings, parameter counts, source hashes and training metadata |
evaluation-val.json |
Uploaded-pair validation metrics |
evaluation-test.json |
Uploaded-pair test metrics |
evaluation-challenge.json |
Uploaded-pair challenge metrics |
predict.py |
Minimal runnable inference example |
finetuning/ |
Bundled custom model and adapter source |
manifest.json |
File integrity manifest |
How to get started
The release bundles the exact base, tokenizer, adapter and compatible PyTorch source. Do not try AutoModel.from_pretrained() or PeftModel.from_pretrained(): those are not this implementation's loaders.
Install Python 3.12.12 and the repository's locked CPU dependencies:
git clone https://github.com/nehalvaghasiya/transformers-from-scratch.git cd transformers-from-scratch git checkout 5fdb74437670658c928e1f0fa4dacf80fca9c0e1 uv python install 3.12.12 uv sync --locked --extra cpu --group pretraining --group demoWith the
hfCLI available onPATH, download the whole model repository into a sibling directory:hf download nehal-vaghasiya/tinysql-lora \ --local-dir ../tinysql-lora-bundleRun the included inference script using the locked environment:
uv run --no-sync python ../tinysql-lora-bundle/predict.py \ "how many orders are completed"
The script imports the bundled finetuning source from the model folder, loads base/model.pt, attaches adapter.pt, and generates the SQL continuation for the supplied question. The exact output depends on the prompt; do not treat an illustrative SQL example as a verified prediction for unseen text.
A no-installation base vs. LoRA Space illustrates both models and runs generated SQL against the isolated sample database.
Training details
Training data
The dataset is created by finetuning.tinysql.data using seed 42, with synthetic natural-language questions mapped to SQL for the fixed three-table schema.
| Split | Rows | Purpose |
|---|---|---|
| Original base training | 6,400 | Fit the custom base model |
| Validation | 320 | Select base and adapter checkpoints |
| Historical test | 320 | Exploratory evaluation |
| Challenge | 160 | Unfamiliar synthetic paraphrase stress test |
| Augmented LoRA training | 15,930 | Paraphrased training-only examples for adapter optimization |
The base training set includes 1,968 distinct questions among 6,400 rows; the augmented adapter dataset includes 3,525 distinct questions among 15,930 rows. These are not counts of independent unique examples. No exact natural-language training questions overlap held-out splits; however, SQL skeletons and output programs are repeated across splits. Validation/test/challenge are not tests of unseen schemas or unseen classes of SQL programs.
Preprocessing and objective
- Fit the custom 174-token vocabulary on the original training data only.
- Format input as a natural-language question with fixed-schema context and a target SQL continuation.
- Train with autoregressive cross-entropy applied to SQL answer tokens and EOS only; mask prompt/context positions.
- During adapter training, freeze every base parameter and insert trainable low-rank updates into
qkv,proj,ff.0, andff.2projections. - Decode greedily, with a maximum 85 new SQL tokens and a stop at the original 160-token context boundary.
Base-model training - published paired base
| Setting | Value |
|---|---|
| Base size | 439,552 parameters |
| Layers / attention heads | 2 / 4 |
| Hidden width / context | 128 / 160 |
| Token vocabulary | 174 |
| Dropout | 0.0 |
| Random seed | 42 |
| Optimizer | AdamW, with 5% warmup and cosine decay to 10% of the initial LR |
| Learning rate / weight decay | 0.001 / 0.01 |
| Batch size | 64 |
| Epochs / completed steps | 20 / 2,000 |
| Selected checkpoint | Epoch 15, validation SQL-token CE with a 1e-4 improvement threshold |
| Best validation SQL-token CE | 0.02408 |
| Recorded device / threads | CPU / 2 |
| Recorded training duration | 297.81 s |
LoRA fine-tuning - published adapter
| Setting | Value |
|---|---|
LoRA rank (r) |
8 |
LoRA scaling (alpha) |
16.0 |
| LoRA dropout | 0.0 |
| Target modules | qkv, proj, ff.0, ff.2 |
| Frozen base parameters | 439,552 |
| Trainable adapter parameters | 32,768 |
| Combined base + adapter parameters | 472,320 |
| Adapter/base parameter fraction | 7.45% |
| Adapter/total parameter fraction | 6.94% |
| Random seed | 42 |
| Learning rate / weight decay | 3e-4 / 0.0 |
| Batch size | 64 |
| Epochs / completed steps | 1 / 90 |
| Checkpoint selection | Minimum validation SQL-token cross-entropy |
| Base CE → adapter CE | 0.02408 → 0.01617 |
| Recorded device / threads | CPU / 2 |
| Recorded LoRA training duration | 14.45 s |
The base trainer only replaces its best checkpoint when validation loss improves by more than 1e-4. Consequently, base/history.json reports an epoch-20 loss of 0.02407196, marginally below the saved epoch-15 checkpoint's 0.02407979, but the difference is smaller than the trainer's improvement threshold. The published checkpoint and score correspond to epoch 15, as stated in base/metadata.json.
This is custom LoRA implemented in PyTorch, not PEFT. No adapter_config.json/PEFT loader is implied.
Evaluation
Testing data, factors and metrics
Published full-split evaluation covers 320 validation, 320 test and 160 challenge rows. The model generates SQL autoregressively and runs candidate and gold programs on two deterministic populated SQLite databases, seeded 2026 and 2027. Its evaluation tracks:
- Execution accuracy: Fraction where the generated query returns the gold result on the evaluation databases.
- Exact match: Fraction matching the normalized target SQL string under the project's evaluation rules.
- SQL validity: Fraction that parses/executes successfully under the project's SQLite validity check.
- Family-specific accuracy: Results for counts, lookups, aggregates, joins, state filters and ranking templates.
- Paired gain: Row-matched LoRA minus frozen-base execution accuracy, with a paired bootstrap interval; this does not measure run-to-run variability.
Results for the uploaded base + adapter
| System | Validation (n=320) | Test (n=320) | Challenge (n=160) |
|---|---|---|---|
| Frozen paired base | 80.62% | 78.75% | 22.50% |
| Uploaded rank-8 LoRA | 83.12% | 89.06% | 21.88% |
| Word TF-IDF retrieval | 50.31% | 33.75% | 41.25% |
| Character TF-IDF retrieval | 65.31% | 52.81% | 69.38% |
| Word TF-IDF + slot copying | 78.44% | 64.06% | 78.12% |
| LoRA − frozen base | +2.50 pp | +10.31 pp | −0.62 pp |
Accuracy values are rounded to two decimal places. Source exact values are fractions of 320/160 examples.
Additional metrics for the uploaded weights
| Metric | Validation | Test | Challenge |
|---|---|---|---|
| Base SQL validity | 88.12% | 80.00% | 40.62% |
| LoRA SQL validity | 88.75% | 90.31% | 38.75% |
| LoRA exact match | 83.12% | 89.06% | 21.88% |
| Paired execution-accuracy gain (pp) | +2.50 | +10.31 | −0.62 |
| Row-paired bootstrap 95% interval (pp) | [+0.94, +4.38] | [+7.19, +13.75] | [−1.88, 0.00] |
Published matching evidence: evaluation-val.json, evaluation-test.json and evaluation-challenge.json. These files identify the exact base, tokenizer and adapter hashes listed in this card. These values were read from the uploader's recorded evaluation files; they were not independently re-executed for this documentation.
Historical project results - not these uploaded weights
The original GitHub experiment reported a separately managed base_small / aug_r8_fast pair with the following execution scores:
| Historical system | Validation | Historical test | Challenge |
|---|---|---|---|
| Frozen historical base | 69.38% | 73.44% | 22.50% |
| Historical rank-8 LoRA | 75.94% | 88.75% | 22.50% |
These figures belong to different checkpoints and should not be substituted for the published model's 80.62/78.75/22.50% base and 83.12/89.06/21.88% LoRA results. The historical run's test/challenge splits were previously inspected, and the complete rationale for choosing its rank-8 configuration over alternatives was not fully recorded. See the GitHub experiment report.
Summary and failure analysis
The released LoRA improves the paired base on familiar synthetic validation and test paraphrases, especially within some state-order/count templates, but does not generalize to the challenge phrasing. On challenge, execution accuracy is slightly worse than the base and well below the non-neural TF-IDF + slot-copying baseline. As a result, this model is principally a reproducible demonstration of from-scratch model development and LoRA adaptation, rather than evidence of robust text-to-SQL performance.
Technical specifications
- Architecture: Decoder-only Transformer with two causal self-attention layers, hidden size 128, four heads, tied token/output embeddings, 174-token custom vocabulary and 160-token context.
- Training objectives: SQL-continuation next-token cross-entropy, first for the full randomly initialized base and then for trainable LoRA updates with the base frozen.
- LoRA implementation: Custom tensor updates to attention and feed-forward projections; rank 8, scaling alpha 16, no dropout.
- Compute: CPU; two threads. GPU / accelerator performance, GPU memory and production service throughput not established.
- Software recorded with this release: Python 3.12.12, PyTorch 2.8.0+cpu, NumPy 2.5.3, Linux x86-64 WSL2. The source dependencies are locked in
uv.lock. - Transformers / PEFT versions: Not applicable to this custom implementation. The adapter is not an installable PEFT-format checkpoint.
Reproduce training and evaluation
Using the source repository and locked dependencies described above, run from finetuning/. This trains a new independent pair; its metrics must be measured rather than assumed to match the uploaded checkpoint:
cd finetuning
uv run --project .. --no-sync python -m finetuning.tinysql.data --out data
uv run --project .. --no-sync python -m finetuning.lora_sql.augment \
--src data/train.jsonl --out runs/augmentation/train.jsonl
uv run --project .. --no-sync python -m finetuning.tinysql.train \
--data-dir data --out runs/base --epochs 20 --seed 42 \
--device cpu --num-threads 2
uv run --project .. --no-sync python -m finetuning.lora_sql.train_lora \
--base runs/base --data runs/augmentation --out runs/adapter \
--rank 8 --alpha 16 --lr 0.0003 --batch 64 \
--epochs 1 --steps 90 --seed 42 --device cpu
uv run --project .. --no-sync python -m finetuning.lora_sql.evaluate_lora \
--base runs/base --data data --adapter runs/adapter/adapter.pt \
--split test --count 0 --output runs/evaluation/test.json
The documented --count 0 evaluates the entire split. Use new output paths to avoid overwriting earlier experiments. For integrity details, refer to the dataset card and the reproduction summary.
Citation
This is a model/software release, not a separately published research paper. To credit the work:
@misc{vaghasiya2026tinysql,
author = {Vaghasiya, Nehal},
title = {TinySQL: LoRA Fine-Tuning of a Transformer Built from Scratch},
year = {2026},
howpublished = {Hugging Face model repository},
url = {https://huggingface.co/nehal-vaghasiya/tinysql-lora},
note = {Source code: https://github.com/nehalvaghasiya/transformers-from-scratch}
}
APA: Vaghasiya, N. (2026). TinySQL: LoRA Fine-Tuning of a Transformer Built from Scratch [Machine learning model]. Hugging Face. https://huggingface.co/nehal-vaghasiya/tinysql-lora
Underlying technique: Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models. This credits the approach, not the authorship of this particular code or checkpoint.
Model card authors and contact
Maintainer: Nehal Vaghasiya. Please use the model's Community tab or the GitHub project to discuss issues, report discrepancies or request clarifications.