sinlib: Pretrained Sinhala NLP and Spellchecking Models

This repository hosts the official pretrained model checkpoints, vocabularies, and statistical datasets for the sinlib library—a comprehensive Sinhala NLP toolkit.

Models & Artifacts Hosted

This repository contains the following files loaded dynamically by sinlib.spellcheck.TypoDetector:

• bigru_detector.pt: A bidirectional GRU sequence labeling model (BiGRUSequenceLabeler) trained to detect spelling errors and character substitutions at the akshara level.
• bigru_corrector.pt: A sequence-to-sequence bidirectional GRU encoder-decoder model with Attention (BiGRUSeq2Seq) that performs generative character/akshara corrections.
• akshara_vocab.json: Vocabulary mappings mapping Sinhala phonological units (aksharas) and basic punctuation to token IDs for the neural models.
• akshara_ngram.json: Stored counts and vocabularies used by the statistical Trigram model (AksharaNGram) for likelihood evaluation.
• news_unigrams.json & news_bigrams.json: Pre-calculated unigram and bigram word-level frequencies extracted from Sinhala news corpora, used for context-aware candidate re-ranking (Stupid Backoff).
• dictionary.npy: The canonical baseline dictionary containing ~61K valid Sinhala words.
• ngram_probs.npy: N-gram probabilities used by the default PreTrainedTokenizer fallback checks.

Additional models, configuration files, and vocabulary resources required by the sinlib package are also included in this repository.

---

Intended Use

These weights are designed to be loaded directly through the sinlib Python package.

Installation

pip install sinlib

Example Inference (Spellcheck)

from sinlib.spellcheck import TypoDetector

# Automatically downloads and caches the model files from this HF repository
detector = TypoDetector.from_pretrained("Ransaka/sinlib")

# Run spellcheck (with punctuation preservation & unicode normalization)
sentence = "කොළඹ වරායේ සිට බස්නහිර දෙසින් නාවික සැතපුම් 17ක් පමණ දුරන්."
corrected = detector(sentence)

print(corrected)
# Output: "කොළඹ වරායේ සිට බස්නාහිර දෙසින් නාවික සැතපුම් 17ක් පමණ දුරින්."

Typo Detection Fallback

If PyTorch is not installed in the target environment, sinlib falls back automatically to the statistical N-Gram model (akshara_ngram.json) to perform spelling checks.

# Check word suspicion level
is_typo = detector.is_word_suspicious("පසලට") # True

For more details, usage examples, and API references, please refer to the official documentation:

https://sinlib.readthedocs.io/en/latest/