BERT Base Uncased LoRA - Shakespeare Q&A

This model is a LoRA (Low-Rank Adaptation) fine-tuned version of BERT Base Uncased for extractive question answering on Shakespeare's works. It specializes in answering questions about characters, plots, themes, and literary elements in Shakespeare's plays and sonnets.

Model Description

• Model type: Question Answering (Extractive)
• Base model: bert-base-uncased
• Fine-tuning method: LoRA (Low-Rank Adaptation)
• Domain: Shakespeare's literary works
• Language: English (Early Modern English / Shakespearean)
• Library: PEFT

Intended uses & limitations

Intended uses

• 🎓 Educational tools for Shakespeare studies
• 📚 Literature analysis and research assistance
• 👨‍🎓 Student homework help for Shakespeare courses
• 🔬 Digital humanities research projects
• 🤖 Chatbots focused on classical literature
• 📖 Reading comprehension for Shakespeare texts

Limitations

• Domain-specific: Optimized for Shakespeare only; may not work well on modern texts
• Extractive only: Cannot generate answers not present in the provided context
• Context length: Limited to 512 tokens (BERT's maximum sequence length)
• Language style: Best performance with Shakespearean/Early Modern English
• No real-time knowledge: Cannot answer questions about events after training

How to use

Quick start

from transformers import BertTokenizerFast, BertForQuestionAnswering
from peft import PeftModel
import torch

# Load the model and tokenizer
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
base_model = BertForQuestionAnswering.from_pretrained("bert-base-uncased")
model = PeftModel.from_pretrained(base_model, "Hananguyen12/bert-base-uncase-lora-shakespeare-plays")

def answer_question(question, context):
    inputs = tokenizer(question, context, return_tensors="pt", max_length=512, truncation=True)
    
    with torch.no_grad():
        outputs = model(**inputs)
    
    start_idx = torch.argmax(outputs.start_logits)
    end_idx = torch.argmax(outputs.end_logits)
    
    answer_tokens = inputs['input_ids'][0][start_idx:end_idx+1]
    answer = tokenizer.decode(answer_tokens, skip_special_tokens=True)
    
    return answer

# Example usage
question = "Who is Romeo?"
context = "Romeo Montague is a young man from the Montague family in Verona. He falls in love with Juliet Capulet."
answer = answer_question(question, context)
print(f"Answer: {answer}")

Batch processing

def batch_answer_questions(questions, contexts, batch_size=8):
    results = []
    
    for i in range(0, len(questions), batch_size):
        batch_q = questions[i:i+batch_size]
        batch_c = contexts[i:i+batch_size]
        
        inputs = tokenizer(batch_q, batch_c, return_tensors="pt", padding=True, truncation=True, max_length=512)
        
        with torch.no_grad():
            outputs = model(**inputs)
        
        for j in range(len(batch_q)):
            start_idx = torch.argmax(outputs.start_logits[j])
            end_idx = torch.argmax(outputs.end_logits[j])
            
            answer_tokens = inputs['input_ids'][j][start_idx:end_idx+1]
            answer = tokenizer.decode(answer_tokens, skip_special_tokens=True)
            results.append(answer)
    
    return results

Training details

Training data

The model was fine-tuned on a comprehensive Shakespeare dataset containing:

• Size: ~15,000+ question-answer pairs
• Coverage: Major plays (Hamlet, Romeo & Juliet, Macbeth, Othello, King Lear, etc.)
• Question types:
  • Character analysis (25%)
  • Plot understanding (30%)
  • Thematic interpretation (20%)
  • Language/literary analysis (15%)
  • Historical context (10%)

Training procedure

LoRA configuration

• Rank (r): 16
• Alpha: 32
• Dropout: 0.1
• Target modules: ["query", "key", "value", "dense"]
• Trainable parameters: ~0.3% of total model parameters

Training hyperparameters

• Base model: bert-base-uncased
• Task: Extractive Question Answering
• Optimizer: AdamW
• Learning rate: 2e-4
• Batch size: 16 (effective with gradient accumulation)
• Max sequence length: 512
• Epochs: 4
• Warmup steps: 500
• Weight decay: 0.01

Compute infrastructure

• Hardware: NVIDIA Tesla T4/V100 GPU
• Software: PyTorch, Transformers, PEFT
• Training time: ~2-3 hours
• Memory usage: ~12GB GPU memory

Evaluation

Metrics

The model achieves strong performance on Shakespeare-specific question answering:

Performance by question type

Example applications

Educational chatbot

class ShakespeareChatbot:
    def __init__(self):
        self.tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
        base_model = BertForQuestionAnswering.from_pretrained("bert-base-uncased")
        self.model = PeftModel.from_pretrained(base_model, "Hananguyen12/bert-base-uncase-lora-shakespeare-plays")
    
    def ask(self, question, play_context):
        return answer_question(question, play_context)

# Usage
chatbot = ShakespeareChatbot()
answer = chatbot.ask("What motivates Lady Macbeth?", macbeth_context)

Literature analysis tool

def analyze_character(character_name, context_passages):
    questions = [
        f"Who is {character_name}?",
        f"What motivates {character_name}?",
        f"How does {character_name} change throughout the play?",
        f"What is {character_name}'s relationship to other characters?"
    ]
    
    analysis = {}
    for question in questions:
        for passage in context_passages:
            answer = answer_question(question, passage)
            if answer and len(answer.strip()) > 3:
                analysis[question] = answer
                break
    
    return analysis

Environmental impact

• Hardware type: NVIDIA Tesla T4/V100
• Hours used: ~3 hours total training time
• Cloud provider: Google Colab
• Carbon emitted: Minimal due to efficient LoRA training

Technical specifications

Model architecture

• Base model: BERT Base Uncased (110M parameters)
• LoRA adaptation: 16-rank adaptation on attention layers
• Total parameters: ~110M (only ~0.3% trainable)
• Model size: ~440MB (base) + ~2MB (LoRA adapter)

Software versions

• Transformers: 4.35.0+
• PEFT: 0.6.0+
• PyTorch: 2.0.0+
• Python: 3.8+

Citation

@misc{shakespeare-bert-lora-2025,
  title={BERT Base Uncased LoRA - Shakespeare Q&A},
  author={Hananguyen12},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/Hananguyen12/bert-base-uncase-lora-shakespeare-plays},
  note={LoRA fine-tuned BERT model for Shakespeare question answering}
}

Model card authors

Hananguyen12

Model card contact

For questions about this model, please open an issue in the model repository or contact through Hugging Face.

License

This model is released under the MIT License. The base BERT model follows its original Apache 2.0 license.

Acknowledgments

• Base model: Google's BERT Base Uncased
• LoRA technique: Microsoft's Low-Rank Adaptation
• Framework: HuggingFace Transformers and PEFT
• Training platform: Google Colab
• Dataset: Shakespeare's complete works

---

"All the world's a stage, and all the men and women merely players." - As You Like It, Act II, Scene VII