Whisper Small - Korean to English Audio Translator (V2 Cross-Attention Optimized)

This repository hosts an updated, fine-tuned version of openai/whisper-small engineered to translate spoken Korean audio directly into English text transcripts.

Unlike V1 which only adapted self-attention mechanisms, this V2 model unfreezes encoder-decoder cross-attention mapping layers, resulting in a 3x translation baseline improvement and successful alignment of domain-specific sports terminology (e.g., baseball pitching terminology like slider and splitter).

Model Evaluation Performance

• V2 Cross-Attention Fine-Tuned Model BLEU: 14.73
• Original Baseline Whisper-Small BLEU: 4.64

Training Hyperparameters & Infrastructure

• Hardware: NVIDIA A100 GPU
• Quantization: 4-bit NormalFloat (nf4) with double quantization
• LoRA Configurations: Rank ((r)) = 32, Alpha ((\alpha)) = 64
• Target Modules: ["q_proj", "v_proj", "out_proj", "k_proj", "fc1", "fc2"] (Cross-Attention Enabled)
• Learning Rate: 5e-5 (Stabilized)
• Precision: Native Mixed Precision (bf16)
• Optimization Batching: Effective Batch Size = 32 (Per Device Train Batch = 8, Gradient Accumulation Steps = 4)