🧪 Experimental v1, not production-ready yet. It works: answers stay in pure Hindi with no English words in brackets, and it follows requested list counts. It also has known issues (below), and a v2 is in progress. In my 25-prompt evaluation, base Gemma 4 12B with a short Hindi system prompt scored higher, so use that setup if you need the best quality today.
Part of my weekly 🇮🇳 Hindi LLM Series.
v1 of my Gemma 4 12B Hindi fine-tune, trained on 12.5k Hindi instruction pairs. It meets its main goals: no English words in brackets, and requested counts followed (7/7). Because much of the training data is machine-translated, it also picked up some translation habits: extra length, a few artifacts, and occasional factual slips. A 4-line system prompt on the base model currently scores higher (24 vs 20 fully correct out of 25). v2 will fix this by training on the model's own clean Hindi answers (self-distillation).
| Base + system prompt | This LoRA (temp 0.3) | |
|---|---|---|
| Manual review: correct / partly wrong / wrong | 24 / 1 / 0 | 20 / 3 / 2 |
| English words in brackets | 0 | 0 |
| Requested list counts met | 7/7 | 7/7 |
| Visarga used as a colon ("हैः") | 2 | 45 |
| Average answer length | 219 chars | 628 chars |
| Chatbot filler ("नमस्ते! मुझे खुशी है…") | 0 | 5 answers |
Most of the training data is machine-translated into Hindi, and the 12B base already writes better Hindi than that data. So v1 picked up some of the data's habits:
The same data improved my smaller models (0.6B–4B). At 12B, the base model is already stronger than the data.
Base Gemma 4 12B with this system prompt:
हमेशा केवल शुद्ध हिंदी में उत्तर दें। कोष्ठक में अंग्रेज़ी शब्द न लिखें।
जितना पूछा जाए, ठीक उतना ही दें। Markdown का उपयोग न करें।
गणित या तर्क के प्रश्नों में अंतिम उत्तर से पहले एक-दो पंक्तियों में हल दिखाएँ।
The last line fixes the base model being too terse on maths. It was added after the evaluation.
The adapter is in the last-checkpoint folder.
from huggingface_hub import snapshot_download
from unsloth import FastModel
snapshot_download("pankajpandey-dev/gemma-4-12b-hindi-instruct-checkpoints",
allow_patterns=["last-checkpoint/*"], local_dir="ckpt")
model, tokenizer = FastModel.from_pretrained("ckpt/last-checkpoint",
max_seq_length=2048, load_in_4bit=True)
Recommended settings: temperature 0.3 or lower (at 1.0 it can produce foreign-script tokens), max new tokens around 400. Review outputs before using them in anything user-facing.
| Base | unsloth/gemma-4-12b-it (Gemma 4 12B instruct), 4-bit QLoRA |
| Data | AI4Bharat indic-instruct-data-v0.1: anudesh Hindi (7,577) + 4,000 dolly Hindi; Cohere Aya Hindi (1,153). 12,478 examples after removing those over 1,024 tokens |
| LoRA | r=16, alpha=16, dropout 0, attention + MLP layers, vision layers frozen; 65.6M trainable parameters (0.55%) |
| Loss | Response-only (assistant tokens only), Gemma 4 chat template with empty thought channel |
| Optimiser | AdamW 8-bit, LR 1e-4, cosine, 20 warmup steps, weight decay 0.01 |
| Batch | 1 × 8 gradient accumulation, max length 1,024 |
| Schedule | 1 epoch, 1,560 steps |
| Precision | float32 compute (Gemma 4 can't train in fp16; the T4 has no bf16) |
| Hardware | Free Kaggle T4, 9 h 41 min |
| Final loss | 2.60 → ~1.2 plateau; 1.23 average over the run |
v2 with self-distillation: train on the 12B's own clean Hindi answers (base + system prompt, filtered) instead of translated data, so the pure-Hindi behaviour is built in without losing quality.
Built with Unsloth and TRL. Data by AI4Bharat and Cohere Aya. 🙏
adapter_model.safetensors 262 MBlast-checkpoint/adapter_model.safetensors 262 MBlast-checkpoint/optimizer.pt 134 MBlast-checkpoint/rng_state.pth 14.6 kBlast-checkpoint/training_args.bin 6.2 kBtraining_args.bin 6.2 kBlast-checkpoint/scheduler.pt 1.5 kBgoogle/gemma-4-12B-it