harsha-desaraju/telugu-llama-small

📌 Model Description

Telugu-Llama (64M) is a decoder-only transformer model based on the LLaMA architecture, trained from scratch on ~579M Telugu tokens. The model is designed to capture linguistic structure across a diverse mix of Telugu sources including classical literature, poetry, and modern content.

• Architecture: LLaMA-style causal language model
• Parameters: ~64M
• Language: Telugu
• Training Tokens: ~579 million
• Training Epochs: 2
• Final Eval Loss: 3.033

---

🧠 Model Architecture

LlamaConfig(
    vocab_size=16384,
    hidden_size=512,
    intermediate_size=2048,
    num_hidden_layers=12,
    num_attention_heads=8,
    num_key_value_heads=4,
    max_position_embeddings=2048,
    rms_norm_eps=1e-5,
    rope_theta=10000.0,
    hidden_act='silu'
)

---

🔤 Tokenizer

• Type: Byte Pair Encoding (BPE)
• Vocab Size: 16,384
• Training Data: Same domain as model training corpus
• Token Fertility: 1.9

This tokenizer is optimized for Telugu morphology and script efficiency.

---

📚 Training Data

The model was trained on a curated and diverse Telugu corpus:

Sources

• ai4bharat-sangraha/verified/tel

• Custom scraped datasets including:

  • Telugu poetry (classical & modern)
  • Ancient music literature
  • Film songs and lyrics
  • General web text

---

⚙️ Training Details

Training Configuration

TrainingArguments(
    num_train_epochs=2,
    per_device_train_batch_size=16,
    gradient_accumulation_steps=4,
    learning_rate=1e-4,
    weight_decay=0.1,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    max_grad_norm=1.0,
    fp16=True
)

---

📊 Evaluation

• Metric: Cross-entropy loss
• Final Eval Loss: 3.033
• Perplexity: 20.76

---

Example

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("harsha-desaraju/telugu-llama-small")
model = AutoModelForCausalLM.from_pretrained("harsha-desaraju/telugu-llama-small")

inputs = tokenizer("ఎందరో మహానుభావులు అందరికి ", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))

---

⚠️ Limitations

• Small model size (64M) limits:

  • Long-range reasoning
  • Factual accuracy

• May produce:

  • Hallucinations
  • Repetitive text

• Domain bias toward:

  • Literary and cultural text

• No instruction-following ability (just a text completion model)

---

⚖️ Bias & Risks

• Training data includes scraped content → potential:

  • Cultural bias
  • Stylistic skew toward poetry/lyrics

• May reflect biases present in Telugu web and literary corpora

Use cautiously in sensitive applications.