BashkirRoBERTa Token LID

Experimental ONNX word-level language identification for mixed Bashkir, Russian and Tatar messages.

Overview

This model labels each word and punctuation token in a message. It can retain language switches inside a sentence, including quoted speech, English inserts and inflected borrowed words. It includes a standalone CPU adapter, FP32 and dynamic INT8 graphs. INT8 is the default. The encoder and classification head were fine-tuned together for token-level language discrimination on conversational and code-switched text.

Contents

Files and Configurations

Training texts, annotation batches, evaluation corpora, PyTorch checkpoints and development logs are not included.

Model Architecture

Examples

These outputs were generated by the bundled INT8 runtime. O tokens are omitted here for readability.

BA means Bashkir; RU Russian; TAT Tatar; OTHER other languages, Latin inserts, brands or code; O punctuation, standalone numbers and symbols. Borrowed stems with Bashkir grammar have target label BA under the current rubric. Shared BA/TAT words can be ambiguous.

Method

BashkirRoBERTa base encoder
  ├── 1. tokenization with word-level subtoken alignment
  ├── 2. fine-tune token classification head on mixed BA/RU/TAT corpus
  ├── 3. export FP32 ONNX inference graph
  └── 4. dynamic INT8 quantization & sliding-window inference

A pretrained Bashkir-language encoder was adapted for word-level multilingual tagging, then further fine-tuned with conversational annotations. Training includes multilingual fragments, synthetic code-switching and reviewed labels. Ambiguous words are masked during supervision; contextual variants of each new situation remain in the same training or validation partition. FP32 was exported from the selected checkpoint and dynamically quantized to INT8. The adapter preserves the original first-subword alignment. Long messages use overlapping complete-word windows and retain their tail. A single word that cannot fit in one window raises an explicit error.

Benchmark

The following figures use one complete message per CPU invocation. Strict lexical accuracy excludes neutral punctuation and standalone numbers. Exact messages require all word tokens in the message to match the reference tags.

Validation annotations were used for epoch selection and are reported separately in benchmark_summary.json. The suites above assess realistic multilingual conversational settings:

• Conversational test suite: standard mixed Bashkir/Russian/Tatar conversational sentences.
• Code-switching & loanwords: rapid language transitions, Russian borrowings with Bashkir grammar, and foreign words.
• Everyday dialogue & chat: natural colloquial dialogue messages, informal questions, and responses.

Quality and Use

Use this release for exploratory annotation, language-span analysis and routing mixed-language messages. Send uncertain cases to review. Confidence is an uncalibrated model score. A majority vote over word tags can provide a coarse sentence label, but hides minority-language inserts and is not a separate trained classifier.

Limitations

• Curated conversational test suites; recommended for routing, span analysis, and dataset cleaning.
• Bashkir without its distinctive letters can be confused with Tatar; some forms are intrinsically ambiguous.
• Russian-root words with Bashkir endings, shared Turkic words and short switches can be mislabeled.
• High confidence can accompany errors. OTHER is a broad fallback rather than a complete language inventory.
• Overlapping windows preserve coverage but cannot guarantee correct classification across long contexts.
• Public runtime uses ONNX Runtime; the standard Transformers token-classification pipeline is not provided.

Related Resources

• Bashkir LID (Binary) — fast binary language gate (ba vs non_ba) for corpus pre-filtering and OCR triage.
• Bashkir Multiclass LID — whole-text 4-way language classifier (ba, tt, ru, other) when token-level spans are not needed.
• BashkirRoBERTa — the base masked-language encoder for other adaptation tasks.

Usage

pip install onnxruntime sentencepiece numpy huggingface_hub

import importlib.util
from pathlib import Path
from huggingface_hub import snapshot_download

directory=Path(snapshot_download("failed09/bashkir-roberta-token-lid", allow_patterns=[
    "onnx/model_int8.onnx", "spm_bashkir_bert_16k.model", "config.json", "META.json", "runtime.py"]))
spec=importlib.util.spec_from_file_location("bashkir_token_lid_runtime",directory/"runtime.py")
runtime=importlib.util.module_from_spec(spec)
spec.loader.exec_module(runtime)
lid=runtime.BashkirTokenLID(directory)
for token in lid.predict("Ул миңә see you tomorrow тип әйтте, значит завтра встретимся."):
    print(token)

After downloading, inference runs locally. To use FP32, also download onnx/model_fp32.onnx and pass precision="fp32". Each output token includes its label, confidence and start/end offsets in the original Python Unicode string.

License

The model weights, tokenizer and runtime are distributed under Apache-2.0. The base encoder is also released under Apache-2.0. Rights in source texts remain with their respective owners; original corpora are not redistributed.

Citation

@software{failed09_bashkir_roberta_token_lid_2026,
  title = {BashkirRoBERTa Token LID},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-roberta-token-lid},
  note = {Experimental ONNX word-level language identification}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.