mT5-base-KenSwQuAD-Abstractive

The final model in a 3-stage Hierarchical Curriculum Learning pipeline for Swahili Question Answering.

This model generates abstractive answers to questions about Swahili text, using Context Scaffolding technique with anchor tokens to improve answer localization.

Model Description

Pipeline Position

This model is Stage 3 of a curriculum learning pipeline:

Performance

Evaluation Results

Training Progress

Best model saved at Epoch 5 (highest BERTScore F1)

Usage

Basic Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Load model and tokenizer (MUST be from same checkpoint!)
tokenizer = AutoTokenizer.from_pretrained("kikwaib/mt5-base-kenswquad-abstractive")
model = AutoModelForSeq2SeqLM.from_pretrained("kikwaib/mt5-base-kenswquad-abstractive")

# Simple question answering
question = "Nani aliandika kitabu hiki?"
context = "Kitabu hiki kiliandikwa na Ngugi wa Thiong'o. Kilitolewa mwaka 1967."

input_text = f"question: {question} context: {context}"
inputs = tokenizer(input_text, return_tensors="pt", max_length=1024, truncation=True)
outputs = model.generate(**inputs, max_length=128)
answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(answer)  # "Ngugi wa Thiong'o"

With Context Scaffolding (Recommended)

For optimal performance, use anchor tokens to mark paragraph boundaries:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("kikwaib/mt5-base-kenswquad-abstractive")
model = AutoModelForSeq2SeqLM.from_pretrained("kikwaib/mt5-base-kenswquad-abstractive")

# Apply scaffolding with <p1>, <p2>, ... tokens
question = "Nani aliandika kitabu hiki?"
context = "<p1> Kitabu hiki kiliandikwa na Ngugi wa Thiong'o. <p2> Kilitolewa mwaka 1967."

# Add pointer hint if you know which paragraph contains the answer
input_text = f"question: {question} context: {context} <pointer> <p1>"

inputs = tokenizer(input_text, return_tensors="pt", max_length=1024, truncation=True)
outputs = model.generate(**inputs, max_length=128)
answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(answer)  # "Ngugi wa Thiong'o"

Scaffolding Helper Function

import re

def apply_scaffolding(context, question, paragraph_pointer=None):
    """
    Transform raw context into scaffolded format with anchor tokens.
    
    Args:
        context: Raw text (paragraphs separated by newlines)
        question: The question to answer
        paragraph_pointer: Optional paragraph number (1-indexed) containing the answer
    
    Returns:
        Scaffolded input string
    """
    # Split into paragraphs
    paragraphs = [p.strip() for p in re.split(r'\n+', context) if p.strip()]
    
    # Add anchor tokens
    scaffolded = ""
    for i, para in enumerate(paragraphs):
        p_num = i + 1
        if p_num < 50:
            scaffolded += f"<p{p_num}> {para} "
        else:
            scaffolded += f"{para} "
    
    # Add pointer hint
    hint = ""
    if paragraph_pointer and 1 <= int(paragraph_pointer) < 50:
        hint = f" <pointer> <p{paragraph_pointer}>"
    
    return f"question: {question} context: {scaffolded}{hint}"

# Usage
input_text = apply_scaffolding(
    context="First paragraph.\nSecond paragraph with answer.",
    question="What is in the second paragraph?",
    paragraph_pointer=2
)

Pipeline Usage

from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="kikwaib/mt5-base-kenswquad-abstractive",
    tokenizer="kikwaib/mt5-base-kenswquad-abstractive"
)

result = qa_pipeline("question: Nani aliandika kitabu? context: <p1> Ngugi aliandika kitabu hiki.")
print(result[0]['generated_text'])

Context Scaffolding

What is Context Scaffolding?

Context Scaffolding is a technique that structures input text with special anchor tokens:

1. Paragraph Anchors (<p1>, <p2>, ..., <p49>): Mark paragraph boundaries
2. Pointer Token (<pointer>): Provides a late contextual hint about which paragraph contains relevant information

Why Use Scaffolding?

• Helps the model locate information in long contexts
• Provides attention guidance to relevant paragraphs
• Improves performance on multi-paragraph documents

Example Transformation

Without Scaffolding:

question: What is the capital? context: Kenya is in East Africa. Nairobi is the capital city.

With Scaffolding:

question: What is the capital? context: <p1> Kenya is in East Africa. <p2> Nairobi is the capital city. <pointer> <p2>

Training Details

Training Data

Hyperparameters

Training Infrastructure

Framework Versions

Sample Predictions

Limitations

1. Tokenizer Dependency: Must load tokenizer from this checkpoint (has 50 custom tokens)
2. Context Length: Inputs longer than 1024 tokens are truncated
3. Scaffolding Preference: Best performance requires anchor token formatting
4. Dataset Size: Trained on relatively small abstractive dataset (2,428 samples)
5. Swahili Variants: May struggle with dialectal differences

Ethical Considerations

• Intended Use: Research and educational applications for Swahili NLP
• Not Intended: Production systems without additional validation
• Bias: May reflect biases present in KenSwQuAD training data
• Verification: Answers should be verified for critical applications

Citation

If you use this model, please cite:

@misc{mt5-kenswquad-abstractive,
  author = {Kikwai, B.},
  title = {mT5-base-KenSwQuAD-Abstractive: Hierarchical Curriculum Learning for Swahili QA},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kikwaib/mt5-base-kenswquad-abstractive}
}

Related Models

Acknowledgments

• KenSwQuAD Dataset: For providing Swahili QA pairs
• Google: For the mT5 base model
• Hugging Face: For the Transformers library and model hosting

---

Model trained: December 20, 2025 | Last updated: December 20, 2025