TinyGPT — a Hindi & Hinglish GPT, built from scratch

Two small models from TinyGPT, where every part — the BPE tokenizer, the Transformer, training, decoding, LoRA and the serving code — is written by hand in PyTorch.

Architecture: decoder-only Transformer, Llama recipe (RoPE, RMSNorm, SwiGLU), 6 layers, d_model 384, 6 heads, context 256 tokens, tied embeddings; 8,192-token byte-level BPE with a Devanagari-aware pre-tokeniser.

Training data: 18.2M tokens — Hindi Wikipedia (wikimedia/wikipedia, 20231101.hi, CC BY-SA) and Hinglish sentences (findnitai/english-to-hinglish, Apache-2.0). The parser: the human-annotated part of Hinglish-TOP (Apache-2.0), 2,993 requests; its base model was pretrained without Hinglish-TOP's validation and test requests.

Scores

Use

# pip install torch safetensors "git+https://github.com/TanushLukha/TinyGPT"   (the model's code)
from huggingface_hub import snapshot_download
from tinygpt.serve.engine import TextGenerator, Parser, GenerateParams

path = snapshot_download("tanushlukha/tinygpt")
lm = TextGenerator.from_bundle(f"{path}/lm")
print("".join(p.text for p in lm.stream("भारत की राजधानी", GenerateParams(max_new_tokens=60))
              if hasattr(p, "text")))
print(Parser.from_bundle(f"{path}/parser").parse(["kal subah 7 baje ka alarm laga do"])[0].parse)

Each folder holds model.safetensors, config.json (architecture, task, provenance, scores, SHA-256 of each file — checked on load) and tokenizer.json.

Limitations

A 13.8M-parameter model trained for 53 minutes on a laptop: it writes fluent Hindi but has learned the language, not the facts — names, dates and places in its text are invented (what it knows) — and greedy decoding repeats itself. The parser gets fewer than half of unseen requests exactly right; its most common mistake is a slot missing, extra or mislabelled, and about one mistake in five puts words in a slot that the request doesn't contain. For learning and demos, not for products.