tamizhimet-gemma-2-9b-qlora

QLoRA adapter and classification head for google/gemma-2-9b, fine-tuned to classify a Tamil song-lyric line into one of 9 rhetorical-figure classes (TamizhiMet). This repo holds the LoRA weights and the trained score head; the base model is loaded from its own repo.

Labels: Literal, Uvamai, Uruvagam, Uyarvu_Navirchi_Ani, Thatkurippetra_Ani, Vanchapugalchi_Ani, Madakku, Ilivu, Iyam.

Training

• Data: TamizhiMet train split, 8,680 lines (one label per line).
• Base model loaded in 4-bit NF4 (double quantization, compute dtype fp32) and frozen; LoRA r=16, alpha=32, dropout=0.05 on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj; classification head trained in full.
• 3 epochs, batch 4 x gradient accumulation 4, learning rate 0.0001, cosine schedule with 5% warmup, paged_adamw_8bit, weight decay 0.01, max length 128 tokens.
• Cross-entropy with class weights proportional to n_c^-0.5 (mean 1).
• Seed 42; checkpoint from epoch 3 of 3, the epoch with the best val macro-F1 (val macro-F1 per epoch: 0.355, 0.368, 0.384).

Results

Macro-F1 is averaged over the classes that have gold lines in the split. Val was used to choose the checkpoint, so test is the held-out result.

Per-class F1 (gold support in brackets; fewer than 10 gold lines is unreliable):

Limitations

The training data is dominated by Literal and Uvamai; the rarest classes have very few training lines, and the model seldom or never predicts them (see the per-class table).

Usage (rebuild the classifier exactly as trained)

import torch
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer, BitsAndBytesConfig

repo = "kavienanj/tamizhimet-gemma-2-9b-qlora"
labels = ['Literal', 'Uvamai', 'Uruvagam', 'Uyarvu_Navirchi_Ani', 'Thatkurippetra_Ani', 'Vanchapugalchi_Ani', 'Madakku', 'Ilivu', 'Iyam']
tok = AutoTokenizer.from_pretrained(repo)
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float32)
base = AutoModelForSequenceClassification.from_pretrained(
    "google/gemma-2-9b", num_labels=len(labels), id2label=dict(enumerate(labels)),
    label2id={l: i for i, l in enumerate(labels)}, quantization_config=bnb, device_map={"": 0}, dtype=torch.float32, attn_implementation="eager")
base.config.pad_token_id = tok.pad_token_id
model = PeftModel.from_pretrained(base, repo).eval()

enc = tok("பறவை போல் பறந்தவன்", return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model(**enc, output_hidden_states=True)   # hidden states: one tensor per layer, for probing/steering
print(labels[int(out.logits.argmax(-1))])

tamizhimet_decoder.json in this repo records the base model, quantization, pad token and labels.