Tessera -- span-level multi-label language identification
Labels every span of a document with a set of languages, so low-resource text inside a mixed page survives a corpus filter that would otherwise discard the whole document or file it under one language.
This does not replace GlotLID / OpenLID / OpenLID-v3. They are the comparison, they are the state of the art at document-level LID, and they are Tessera's upstream: their output is an optional prior input. Tessera is one extra pipeline stage.
Model
- 8.68M parameters, 34.7 MB fp32
- 579 labels (479 languages + romanised
*_Latnvariants +und) - Hashed char n-grams (1-5) + word unigrams -> EmbeddingBag; a dense script/shape channel; a document-prior channel; a 1-layer BiGRU over the span sequence with a residual skip; sigmoid multi-label head.
- Trained with the prior zeroed on half of all steps, so one checkpoint serves both the with-prior and without-prior settings.
Results
| system | micro-F1 | macro-F1 | recovery rate |
|---|---|---|---|
| A_fasttext176_document | 0.2803 | 0.0505 | 0.0 |
| B_glotlid_document | 0.3031 | 0.265 | 0.0 |
| C_glotlid_per_line | 0.406 | 0.4788 | 0.1713 |
| D_tessera_no_prior | 0.7712 | 0.6487 | 0.8621 |
| E_tessera_glotlid_prior | 0.7795 | 0.6611 | 0.8396 |
recovery rate = of mid/tail-resource spans in documents a document-level baseline
would have discarded or kept under the wrong language, the fraction correctly
labelled.
Intended use and limits
Intended for corpus curation: deciding which spans of a crawled document to keep and under which language. Not intended as a general-purpose LID API, and not better than GlotLID at labelling a whole document.
Known limits, measured rather than guessed:
- Romanised text remains weak. See the repository's romanised-slice results.
- Above ~80 characters, GlotLID per line is competitive or better. Tessera is built for the short end.
- Passing the document prior slightly lowers the recovery rate even though it raises aggregate F1, because the prior pulls predictions toward the document majority language. If recovery is the goal, do not pass the prior.
- All headline numbers are measured on synthetic data. The reason, and what real data can and cannot show, is documented in the repository.
Ethics
Language identification decides whose text survives a corpus build. Mislabelling a
language removes it from the datasets that later models learn from. The dataset card
lists every covered language with its known weaknesses; tail-language errors are
reported separately rather than averaged away; und is a real label so the model can
decline to guess.
Repository: NagaYu/tessera-span-lid.