Jatayu-hi-11M (जटायु)

A 11.4M-parameter Hindi language model trained from scratch (random initialisation, no pretrained weights) on a single RTX 3060 Laptop GPU with 6 GB VRAM.

Part of the Ramayana Series, small Indic LLMs where each model is named after a figure from the Ramayana. 🌐 Website: https://ashishpatel26.github.io/ramayana-llm/

Named after Jatayu, the vulture who fought Ravana to save Sita despite being far smaller. This is the smallest model in the series: the first proof that the whole pipeline works on a 6 GB GPU.

Model details

Uses

Direct use

Smoke-test / baseline model. Writes grammatical Hindi but invents most facts. For learning and comparison, not for real use.

This is a base model: it continues text, it does not follow instructions or answer questions. Write the beginning of a sentence, not a question.

Downstream use

• Fine-tuning (SFT / LoRA) for Hindi classification, summarisation or instruction following.
• Teaching how pretraining works end to end on consumer hardware.
• A baseline for comparing Hindi tokenizers, data mixes and small-model scaling.

Out-of-scope use

• Any setting where factual accuracy matters: news, medical, legal, financial or educational answers.
• Chat assistants or user-facing products without further tuning and safety work.
• Languages other than Hindi, including romanised Hindi.
• Generating content meant to deceive, impersonate or harass.

Usage

The tokenizer is a SentencePiece model stored in this repo (spm.model). Prompts must start with the language tag <hi>.

# pip install torch transformers sentencepiece huggingface_hub
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import LlamaForCausalLM

repo = "ashishpatel26/jatayu-hi-11m"
model = LlamaForCausalLM.from_pretrained(repo)
sp = spm.SentencePieceProcessor(model_file=hf_hub_download(repo, "spm.model"))

x = torch.tensor([sp.encode("<hi>हिमालय पर्वत")])
y = model.generate(x, max_new_tokens=100, do_sample=True, top_p=0.95,
                   temperature=0.8, repetition_penalty=1.15,
                   eos_token_id=sp.eos_id(), pad_token_id=sp.pad_id())
print(sp.decode(y[0].tolist()))

Sample outputs

Unedited, temperature 0.8, top-p 0.95, repetition penalty 1.15. Prompts: भारत की राजधानी नई दिल्ली, महात्मा गांधी, हिमालय पर्वत, गंगा नदी.

भारत की राजधानी नई दिल्ली का एक क्षेत्र है। यह शहर भारत के लिए सबसे बड़ा राज्य राजधानी बन गया है।

महात्मा गांधी ने राष्ट्रीय फिल्म से शादी की और उनकी सबसे बड़ी फ़िल्म भी निभाई।

हिमालय पर्वत से लेकर दक्षिण और पश्चिम की ओर से लगभग सभी देशों में मिलते हैं, जो देश को उत्तर भारत के लिए विभिन्न दिशाओं में बांटा गया है। इन नदियों द्वारा बहती थी।

गंगा नदी के किनारे स्थित यह नदी बहुत बड़े ही छोटे-छोटे क्षेत्र का निर्माण कर रहा है।

The grammar is correct; the facts are frequently invented (hallucinated).

Comparison within the series

Perplexity measured on 100 batches of held-out Hindi Wikipedia text (every 200th line, never trained on).

Architecture

Llama-style decoder-only transformer (LlamaForCausalLM): RMSNorm, SwiGLU MLP, rotary position embeddings, SDPA attention, tied input/output embeddings.

Training data

• Source: Hindi Wikipedia, wikimedia/wikipedia config 20231101.hi (163k articles, 608 MB raw).
• Cleaning: Unicode NFC, URL removal, lines ≥ 20 characters, ≥ 80% Devanagari characters, exact MD5 de-duplication → 948k lines.
• Split: every 200th line held out for validation → 55.7M training tokens, 0.3M validation tokens.

Tokenizer

SentencePiece Unigram, 32,000 vocab, byte_fallback, character coverage 0.9995, trained on the cleaned corpus. About 1.1 tokens per Hindi word, far fewer than English-centric tokenizers need for Devanagari. Special tokens: <pad>=0, <unk>=1, <s>=2, </s>=3, plus language tags (<hi>, <mr>, …) reserved for future multilingual models.

Training procedure

Full pipeline code (download → clean → tokenizer → pack → train → generate → push): https://github.com/ashishpatel26/Indian-Language-llm-model

Limitations and risks

• Hallucination: at this scale the model learns how Hindi is written, not what is true. Do not use it as a source of facts.
• Narrow domain: trained only on Wikipedia, so it writes in an encyclopaedic style and handles conversational, literary or code-mixed (Hinglish) text poorly.
• No alignment: no instruction tuning, no RLHF, no output safety filtering. It may reproduce biases present in Wikipedia.
• Short context: 256 tokens.

Recommendations

Treat every output as unverified text. Keep a human in the loop, fact-check anything you publish, and fine-tune with curated data before building anything user-facing.

Evaluation

• Metric: cross-entropy loss and perplexity on held-out Hindi Wikipedia (0.3M tokens, 100 random windows of 256 tokens, batch 8).
• Qualitative: sampled continuations for fixed prompts (shown above).
• Not yet evaluated on downstream Hindi benchmarks such as IndicGLUE or IndicXNLI. A base model this small is expected to score near chance on reasoning tasks.

Environmental impact

Technical specifications

• Framework: PyTorch 2.x (CUDA 12.6), Hugging Face Transformers 5.x, SentencePiece 0.2
• Weights: model.safetensors (fp32), config config.json
• Reproducibility: the training script resumes from checkpoints and saves them atomically, so an interrupted run can continue.

Model card authors and contact

ashishpatel26. Questions and issues: https://github.com/ashishpatel26/Indian-Language-llm-model/issues

Citation

@misc{jatayu_hi_11m,
  title  = {Jatayu-hi-11M: a Hindi language model trained from scratch on a 6 GB GPU},
  author = {ashishpatel26},
  year   = {2026},
  url    = {https://huggingface.co/ashishpatel26/jatayu-hi-11m}
}