Bashkir Multiclass LID

Compact ONNX language identification for ba, tt, ru and other, for Bashkir corpus filtering and local CPU inference.

Overview

A compact ONNX language-identification model for Bashkir corpus filtering and local CPU applications. It predicts four classes: ba (Bashkir), tt (Tatar), ru (Russian) and other (other languages, code, URLs, technical noise and corrupted text). It is designed to separate closely related Bashkir and Tatar text while preventing foreign-language and technical material from being silently classified as Bashkir.

Contents

Files and Configurations

Training texts and the original joblib pipeline are not distributed.

Model Architecture

The ONNX graph consumes sparse feature tensors rather than raw strings; the adapter performs tokenization, feature lookup, batching and checksum validation. No custom ONNX operators or Transformers installation are needed.

Examples

Method

The model is a dual-channel linear classifier: TF-IDF char_wb (2–5) and word (1–2) features are fused and passed to an SGDClassifier exported to ONNX. It is trained on Bashkir, Tatar, Russian and other-language material with technical noise. Training texts are not distributed in this repository.

multilingual training corpus (Bashkir, Tatar, Russian & noise)
  ├── 1. extract character n-grams: char_wb (2–5)
  ├── 2. extract word n-grams: words (1–2)
  ├── 3. feature union & TF-IDF weighting
  ├── 4. train multi-class SGDClassifier with calibrated log-loss
  └── 5. export optimized ONNX graph (4-way: ba / tt / ru / other)

Evaluation

The clean multi-source benchmark contains 1,000 examples each from Bashkir, Tatar, Russian and other-language/noise sources; the adversarial set includes closely related Bashkir/Tatar pairs, code-switching, Kazakh, Chuvash, Russian regional text and technical noise.

The benchmark is not expert-reviewed Gold and some examples come from the same source pools used during model development; source overlap was not fully excluded. Exact validation fields are recorded in META.json.

Quality and Use

Mixed-language text does not have a separate class: predictions should be read as the dominant language of the sentence. In a corpus pipeline, confident Bashkir can be retained, while uncertain BA/TT or mixed cases should be sent to a review queue instead of being deleted automatically. Short sentences with common Turkic vocabulary remain intrinsically ambiguous.

Limitations

• A lightweight text classifier, not a universal language detector.
• The reported benchmark is not an expert-reviewed Gold benchmark.
• Confidence values are model scores, not calibrated human probabilities.
• Empty strings, OCR fragments and technical noise should be handled with explicit pipeline rules.

Related Resources

• Bashkir LID (Binary BA vs NON_BA) — a strict ba vs non_ba gate with configurable thresholds; use it when only a pass/reject decision is needed.

Usage

pip install huggingface_hub onnxruntime numpy

import sys
from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    "failed09/bashkir-lid-multiclass",
    allow_patterns=[
        "lid.py", "config.json", "META.json", "model.onnx", "vectorizer.json"
    ],
    revision="main",  # pin a reviewed commit for reproducible deployments
)
sys.path.insert(0, model_dir)
from lid import LanguageIdentifier

lid = LanguageIdentifier(model_dir=model_dir)
texts = [
    "Бары бала бара.",
    "Бүген Казанда яңа мәдәни үзәк ачылды.",
    "В Уфе открыли новый корпус университета.",
    "SELECT id FROM users WHERE active = TRUE;",
]

print(lid.predict(texts).tolist())
print(lid.predict_proba(texts))
print(lid.classes.tolist())  # probability-column order

After downloading, inference runs locally without network access. The probability-column order is always the order in META.json.

License

Distributed under the Apache-2.0 license. This release contains derived model and tokenizer assets, not the source corpus, scans or original document layouts.

Citation

@software{failed09_bashkir_multiclass_lid_2026,
  title = {Bashkir Multiclass LID},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-lid-multiclass},
  note = {Open-source four-class language identification for Bashkir corpus filtering}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.