SindhiFormer — MBAA Architecture Validation
Morpheme-Boundary-Aware Attention (MBAA) for Arabic-Script Low-Resource Language Models
This repository contains the complete validation evidence for SindhiFormer, the first Transformer architecture designed specifically for Sindhi with a novel morpheme-aware attention mechanism.
Key Results
| Metric | MBAA + Mask | Standard Attention |
|---|---|---|
| Perplexity (seed=42) | 245.42 | 245.78 |
| Perplexity (seed=123) | 244.01 | — |
| Perplexity (seed=777) | 245.50 | — |
| Mean ± Std | 244.98 ± 0.69 | 245.78 |
| MBAA Bias Movement | 0.302 (learned ✅) | N/A |
All 3 random seeds beat the standard baseline — the improvement is consistent, not a lucky seed artifact.
Architecture Validated
SindhiFormer (Validation Scale)
├── 6 Layers / 6 Attention Heads / 384 Hidden Dim (~23M params)
├── RoPE (Rotary Position Embeddings)
├── SwiGLU Activation
├── Pre-RMSNorm
├── Weight Tying
└── MBAA: 3 of 6 heads receive morpheme boundary bias (learnable)
What is MBAA?
Standard attention treats all token pairs equally. MBAA adds a learnable negative bias at morpheme boundaries, telling selected attention heads: "tokens within the same word should attend to each other more strongly."
The bias is one scalar per layer — total parameter overhead: 6 floats for the entire model. Yet it produces consistent perplexity improvement because it provides a morphological inductive bias that would otherwise require millions of extra tokens to learn statistically.
MBAA biases learned per-layer patterns:
- Layer 0 (shallowest): bias ≈ -1.00 (barely moved — low-level processing doesn't need morpheme signal)
- Layers 1-3 (middle): bias ≈ -0.70 to -0.82 (strongest movement — morphological composition happens here)
- Layer 5 (deepest): bias ≈ -0.77 (moderate — uses signal for higher-level decisions)
Validation Experiments
Three progressive validation runs were conducted:
| Version | Model | Data | Steps | MBAA Result |
|---|---|---|---|---|
| v1 (baseline) | 4L/256D, 11M | 30K docs | 2000 | ❌ No mask passed — biases frozen |
| v2 (fixed) | 4L/256D, 11M | 30K docs | 2000 | ✅ +0.77% PPL improvement |
| v3 (scaled) | 6L/384D, 23M | 80K docs | 4000 | ✅ +0.33% across 3 seeds |
Production Architecture (Planned)
Based on validation results and scaling law analysis:
SindhiFormer-62M (Production)
├── 16 Layers / 8Q+2KV Heads (GQA) / 512 Hidden Dim
├── SwiGLU (d_ff=1408) / RoPE / Pre-RMSNorm
├── MBAA on 3 of 8 query heads
├── 16,384 vocabulary (morpheme-aware BPE, 1.06 fertility)
└── Training: 4 epochs on 505M tokens (~2B effective), TPU v3-8
Files in This Repository
| File | Description |
|---|---|
SindhiFormer_MBAA_Scaled (1).ipynb |
Primary result — 3 MBAA seeds + 1 standard baseline at 6L/384D scale |
SindhiFormer_MBAA_v2_Fixed (1).ipynb |
Fixed morpheme mask passing — first successful MBAA validation |
SindhiFormer_MBAA_Validation (1).ipynb |
Initial validation (discovered mask-passing bug) |
validation_report.txt |
Formatted validation report |
validation_results.json |
Machine-readable results |
sindhiformer_validation.png |
Training curves visualization |
SindhiFormer_Complete_Guide.md |
Technical handbook covering all concepts |
Related Resources
| Resource | Link |
|---|---|
| SindhiNLTK (NLP toolkit, PyPI) | pypi.org/project/sindhinltk · GitHub |
| Sindhi Corpus 505M | HuggingFace Dataset |
| SindhiLM-Tokenizer-v2 | HuggingFace |
| SindhiLM (GPT-2 prototype) | HuggingFace |
| SindhiLM-Qwen-0.5B-v2 | HuggingFace |
Applicability to Other Languages
This architecture is directly transferable to other Arabic-script, morphologically-rich, low-resource languages:
- Urdu (same script family, similar morphology)
- Pashto (Arabic script, agglutinative)
- Kurdish Sorani (Arabic script)
- Balochi (Arabic script, low-resource)
Replace SindhiNLTK's morpheme rules with the target language's morphological analyzer. The rest of the architecture requires no modification.
Author
Aakash Meghwar — Computational Linguist & Independent Researcher
HuggingFace · GitHub
Citation
@misc{meghwar2026sindhiformer,
author = {Aakash Meghwar},
title = {SindhiFormer: Morpheme-Boundary-Aware Attention for Arabic-Script Language Models},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/aakashMeghwar01/SindhiFormer-Validation}
}