Alpaca Dhivehi Fine-Tuned Flan-T5 with an Extended Tokenizer

This repository contains a fine-tuned Flan-T5 model on the Alpaca Dhivehi dataset, aimed at enabling Dhivehi language instruction-following tasks.

Note: The model can follow instructions and inputs to some extent, but it’s not strictly trained for perfect adherence. Outputs may be partially aligned but are not guaranteed to be fully accurate. Treat results as experimental.

Model Details

• Base model: google/flan-t5-small (or whichever size you used)
• Dataset: Alpaca Dhivehi , Translation from English to Dhivehi
• Training epochs: 5
• Final evaluation:
  • eval_loss: 2.59
  • ROUGE-1: 0.10
  • ROUGE-2: 0.03
  • ROUGE-L: 0.107

Usage

To run inference using the fine-tuned model:

import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

MODEL_PATH = "alakxender/flan-t5-base-alpaca-dv5"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = T5Tokenizer.from_pretrained(MODEL_PATH)
model = T5ForConditionalGeneration.from_pretrained(MODEL_PATH).to(device)

def generate_response(instruction, input_text):
    combined_input = f"{instruction.strip()} {input_text.strip()}" if input_text else instruction.strip()
    inputs = tokenizer(combined_input, return_tensors="pt", truncation=True, max_length=256).to(device)
    
    output_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        num_beams=8,
        repetition_penalty=1.5,
        no_repeat_ngram_size=3,
        do_sample=True,
        early_stopping=True,
        temperature=0.1
    )
    
    decoded_output = tokenizer.decode(output_ids[0], skip_special_tokens=True)
    return decoded_output

# Example usage:
instruction = "ދީފައިވާ މައުޟޫޢާ ބެހޭގޮތުން ކުރު ޕެރެގްރާފެއް ލިޔެލާށެވެ."
input_text = "އިއާދަކުރަނިވި ހަކަތަ ބޭނުންކުރުމުގެ މުހިންމުކަން"
print(generate_response(instruction, input_text))

އިއާދަކުރަނިވި ހަކަތަ ބޭނުންކުރުމުގެ މުހިންމު އެއް މައުޟޫއަކީ ސޯލާ، ވިންޑް، ހައިޑްރޯ، ޖިއޮތަރމަލް، އަދި ހައިޑްރޯއިލެކްޓްރިކް ޕަވަރ ފަދަ އިއާދަކުރަނިވި ހަކަތައިން ގްރީންހައުސް ގޭސްތައް ބޭރުވުން .....

Evaluation Results

From the last evaluation:

{
'eval_loss': 1.947984218597412,
'eval_rouge1': 0.19402897739527525,
'eval_rouge2': 0.13442865174751445,
'eval_rougeL': 0.19137712018920253,
'eval_rougeLsum': 0.192215538761906,
'eval_runtime': 46947.7862,
'eval_samples_per_second': 0.11,
'eval_steps_per_second': 0.11,
'epoch': 10.0
}

Notes

• This fine-tuned model is experimental and intended for research on Dhivehi-language instruction-following tasks.