Kannada GPT-2 Small (kannada-gpt2-32m)

A 31.6M parameter GPT-2 style autoregressive language model trained entirely from scratch on Kannada text. Everything — data pipeline, BPE tokenizer, model weights — built from the ground up on a single NVIDIA RTX 5070.

No pretrained initialization. No fine-tuning. Pure Kannada.

Model Details

Model Description

This is a small GPT-2 model trained from scratch on Kannada text. It uses a custom BPE tokenizer also trained from scratch on the same data. The model can generate coherent Kannada text and produces useful representations for downstream tasks.

• Developed by: AbhiDS16
• Model type: GPT-2 (decoder-only transformer)
• Language: Kannada (kn)
• License: MIT
• Parameters: 31,626,240
• Context length: 512 tokens
• Vocabulary size: 12,000
• Trained from scratch: Yes (no pretrained initialization)

Model Sources

• Repository: https://github.com/thorOdinson16/KanLM
• Demo: Use the Quick Start code below

Uses

Direct Use

The model can be used for:

• Kannada text generation
• Extracting embeddings for downstream tasks (classification, clustering)
• Fine-tuning on task-specific Kannada datasets
• Studying low-resource language model training

Downstream Use

The model's frozen embeddings achieve 73.5% accuracy on Kannada sentiment classification with a simple logistic regression head — demonstrating transferable representations.

Out-of-Scope Use

• Chat/instruction-following (model is not instruction-tuned)
• Production systems requiring high factual accuracy
• Sensitive content generation without safeguards

Bias, Risks, and Limitations

• Small model size: 31.6M parameters limits factual knowledge and reasoning
• Repetition: Tends to repeat phrases in longer generations
• Training data bias: Web text (news, blogs) reflects biases and code-mixing present in online Kannada
• Not instruction-tuned: Raw causal LM — not suitable for chat/QA without fine-tuning
• Data recency: Training data from mC4 (2011–2022)

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("AbhiDS16/kannada-gpt2-32m")
tokenizer = AutoTokenizer.from_pretrained("AbhiDS16/kannada-gpt2-32m")

prompt = "ನಾನು ಇಂದು ಬೆಳಿಗ್ಗೆ"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    temperature=0.7,
    do_sample=True,
    top_p=0.9,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

CulturaX-Kn — 1.35M documents (~4GB) of Kannada web text from mC4. After filtering (Kannada script ratio ≥ 60%, deduplication, length filtering), 12.6M clean sentences were used for training.

Training Procedure

• Precision: fp16 mixed
• Batch size: 16 (effective 32 with gradient accumulation)
• Learning rate: 5e-4 with cosine decay and 1,000 step warmup
• Optimizer: AdamW (β₁=0.9, β₂=0.95, weight decay=0.01)
• Gradient clipping: 1.0
• Epochs: 3
• Total steps: 83,874
• Training tokens: ~463M

Speeds, Sizes, Times

• Hardware: NVIDIA RTX 5070 (8GB VRAM)
• Training time: 7 hours 16 minutes
• Model size on disk: ~126MB (safetensors)
• Throughput: ~3.2 steps/second

Evaluation

Perplexity

Sentiment Classification

Tokenizer Efficiency

Custom BPE tokenizer trained from scratch on Kannada text:

Environmental Impact

• Hardware: NVIDIA RTX 5070 (125W TDP under load)
• Hours used: ~7.3 hours
• Estimated carbon: ~0.35 kg CO2eq (assuming 0.4 kg/kWh grid average)
• Cloud provider: N/A (local desktop)

Technical Specifications

Model Architecture

• 8 transformer layers
• 512 hidden dimension
• 8 attention heads
• 2,048 feed-forward dimension
• GELU activation
• 0.1 dropout

Compute Infrastructure

• GPU: NVIDIA RTX 5070 (8GB VRAM)
• CPU: Intel Core Ultra 9 285H
• RAM: 32GB

Software

• Python 3.10
• PyTorch 2.10
• Transformers 4.x
• Datasets 3.x
• Tokenizers 0.19

Citation

@misc{kannada-gpt2-32m,
  author = {AbhiDS16},
  title = {Kannada GPT-2 Small: A From-Scratch Language Model for Kannada},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/AbhiDS16/kannada-gpt2-32m}},
  note = {Trained entirely from scratch with custom BPE tokenizer}
}

Model Card Contact

Open an issue on GitHub: https://github.com/thorOdinson16/KanLM