Transformer-based English-to-French Translation Model

📌 Overview

This Transformer model was built from scratch as part of a learning project to implement Transformers from the ground up. It was trained on 40 million English-to-French translation examples over 170,000 steps. The goal was to gain a deep understanding of sequence-to-sequence models by implementing every component, including tokenization, embedding layers, positional encoding, encoder-decoder architecture, and inference mechanisms.

🏆 Performance

• Test Accuracy: 0.68

📂 Resources

• Model: Hugging Face
• Dataset: Hugging Face
• Demo App (Gradio): Hugging Face Space
• GitHub Repository: GitHub

🚀 Inference Usage

To use this model for inference, first clone the GitHub repository and install the required dependencies:

git clone https://github.com/ngamcode96/ml-translation-en-fr.git
cd ml-translation-en-fr
pip install -r requirements.txt

Then, you can load the trained weights and deploy an interactive translation app with Gradio:

import torch
from tokenizer import CustomTokenizer
from model import Transformer, TransformerConfig
import gradio as gr

# Load tokenizers
path_to_src_tokenizer = "trained_tokenizers/vocab_en.json"
path_to_tgt_tokenizer = "trained_tokenizers/vocab_fr.json"

src_tokenizer = CustomTokenizer(path_to_vocab=path_to_src_tokenizer)
tgt_tokenizer = CustomTokenizer(path_to_vocab=path_to_tgt_tokenizer)

# Load model
config = TransformerConfig(max_seq_length=512)
model = Transformer(config=config)

path_to_checkpoints = "checkpoints/model.safetensors"
model.load_weights_from_checkpoints(path_to_checkpoints=path_to_checkpoints)
model.eval()

# Translation function
def translate(input_text, skip_special_tokens=True):
    src_ids = torch.tensor(src_tokenizer.encode(input_text)).unsqueeze(0)
    output_ids = model.inference(src_ids=src_ids, tgt_start_id=tgt_tokenizer.bos_token_id, tgt_end_id=tgt_tokenizer.eos_token_id, max_seq_length=512)
    output_tokens = tgt_tokenizer.decode(input=output_ids, skip_special_tokens=skip_special_tokens)
    return output_tokens

# Gradio app for interactive translation
with gr.Blocks() as demo:
    gr.Markdown("## Traduction Anglais → Français")
    
    with gr.Row():
        texte_input = gr.Textbox(label="Texte en anglais", lines=4)
        texte_output = gr.Textbox(label="Texte traduit (Français)", lines=4, interactive=False)
    
    bouton = gr.Button("Traduire")
    bouton.click(translate, inputs=texte_input, outputs=texte_output)

demo.launch()

🎯 Try it now: Use the Hugging Face Space to test the model live!