GitHub | Leaderboard | Dataset | Architecture
License: Apache 2.0 | Authors: Baobab Tech

MillID medium v1

MillID (MultiLingual Language IDentifier) is an open language identification model built by Baobab Tech. It names the language of a passage of text, from a table cell to a paragraph, among 1,641 language-script labels: 1,550 languages written in 56 scripts, and 37 labels for text in a script whose language it cannot name (und_<Script>). MillID medium v1 has 115.0M parameters and an ONNX file of 455 MB.

The middle size, more accurate than tiny v1.5 on both test sets and best run on a GPU; on a laptop CPU it reads about 2 passages a second per thread.

MillID offers:

• Language and script: labels are ISO 639-3 plus ISO 15924 script, so Serbian in Cyrillic (srp_Cyrl) and in Latin (srp_Latn) are told apart. Many of the languages are low-resource, with a few hundred passages each to learn from.
• No tokenizer: it reads UTF-8 bytes, each with its Unicode script and the character n-grams ending there, so any script is input it can read and nothing is an unknown token.
• Short text: a quarter of its training passages were cut to 20 to 80 characters, the length of a heading or a table cell, where language identification is hardest.
• Whole documents: identify_document reads text of any length in windows and returns the document's languages and the segments of each.
• No server, no PyTorch: one ONNX file and a small encoder, in Python (NumPy) or JavaScript (Node and browsers); the two are tested to give the same answers.
• Three sizes, one label set: tiny v1.5, medium v1 and large v1.5 share their labels, so changing size changes no code.

Model overview

Benchmark results

MillID was evaluated on two test sets: test, passages from documents never seen in training, split from the training data by document; and UDHR-LID (udhr), the Universal Declaration of Human Rights labelled by its translators. It is compared with GlotLID and Meta's fastText language identifier (NLLB, 218 labels), each restricted to MillID's labels, its best answer among them: the like-for-like comparison.

Overall results

• test: every language, 1,621 of them. test-common: the languages all three systems know, the head-to-head on equal terms, since fastText cannot answer most of MillID's.
• udhr: the Declaration in the languages MillID knows. udhr-common: in the languages all three know. udhr-all: in every language it has, MillID's unknown ones counted wrong.
• GlotLID's labels came from training on data that includes UDHR, and it labelled the FinePDFs and FineWeb-2 passages in test, so its scores are optimistic on both.

The three MillID models

Macro F1 on test and udhr; this model in bold. All three, language by language, are on the leaderboard.

By passage length

Top-1 accuracy on passages of 20 to 80 characters (short), up to 300 (sentence) and up to 1,000 (paragraph):

MillID on each set

Per language: test

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on 110 languages, worse on 776 and within 0.01 on 735; against fastText, MillID is better on 1616 languages, worse on 0 and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in reports/test/leaderboard.csv.

The 20 languages with the most passages:

Where MillID gains most over GlotLID:

Where MillID loses most to GlotLID:

Per language: udhr

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on 63 languages, worse on 95 and within 0.01 on 200; against fastText, MillID is better on 304 languages, worse on 13 and within 0.01 on 41. Every language, with precision, recall and the unrestricted scores, is in reports/udhr/leaderboard.csv.

The 20 languages with the most passages:

Where MillID gains most over GlotLID:

Where MillID loses most to GlotLID:

Precision, recall (sensitivity), F1 and specificity for every language and system, restricted and as it is, are in reports/<set>/, and the totals in reports/summary.json.

Quick start

Install millid from PyPI, with ONNX Runtime for the CPU ([gpu] for CUDA):

pip install "millid[cpu]"

Identify a passage, several, or a whole document:

from millid import MillID

model = MillID.from_pretrained("baobabtech/millid-medium-v1")
model.predict("Lave men ou ak savon anvan ou manje.", k=3)
# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
model.predict(["one text", "another"], k=3)  # one list of answers per text
model.identify_document(long_text)           # any length: its languages and segments

From the command line:

millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-medium-v1 -k 3

In JavaScript, in Node or a browser, with @baobabtechai/millid from npm:

npm install @baobabtechai/millid onnxruntime-node   # onnxruntime-web in a browser app

import { MillID } from "@baobabtechai/millid";   // "@baobabtechai/millid/web" in a browser, with { ort }
const model = await MillID.fromPretrained("baobabtech/millid-medium-v1");
await model.predict("Lave men ou ak savon anvan ou manje.", 3);

millid.py in this repository is the PyTorch training code, which reads model.safetensors to fine-tune or export.

Best practices

1. Pass the text as it is

The encoder applies Unicode NFKC and collapses whitespace itself. Accents, diacritics and the script are much of what it reads, so do not strip, transliterate or romanise text before passing it.

2. Read long text as a document

predict reads the first 255 bytes of a passage. For anything longer, identify_document reads consecutive windows (at most 256, spread evenly over a longer text), weighs their probabilities by length, and reports the document's languages and the runs of each.

3. Treat a low top probability as unsure

In our tests on real PDFs, a text layer whose characters were mapped to the wrong Unicode code points got a top probability below 0.5 from every MillID model, and reading the same pages from their images gave the right language. A top answer below 0.5 is worth a second look; but see the legacy fonts under Limitations, where that check does not hold.

4. Expect less from very short text

On test, top-1 accuracy is 0.9382 for passages of 20 to 80 characters against 0.9846 for paragraphs. For a heading or a table cell, look at the top three answers rather than the first alone.

5. One text alone and the same text in a batch can score differently

A batch is lower-cased as one string, so a dotted capital İ anywhere in it changes how every text in it is lower-cased (one Greek passage scored 0.976 alone and 0.985 beside Turkish text). The Python and JavaScript runtimes do this identically; send one text at a time where scores must not depend on their neighbours.

6. Choose the size for where it runs

tiny v1.5 on a CPU or in a web page; medium v1 or large v1.5 on a GPU, large v1.5 when accuracy matters most. The three share their labels, so the choice changes no code.

Model data

MillID was trained on passages cut from pages the way a page is read: short lines, sentences and paragraphs.

• PDFs: FinePDFs, PDF text from Common Crawl, labelled by GlotLID.
• Web pages: FineWeb-2, web text from Common Crawl, labelled by GlotLID.
• Storybooks: GlotStoryBook (African Storybook, Pratham Books, LIDA, Little Cree Books) and Nal'ibali's stories in South African languages, labelled by their publishers.
• News in sparse languages: GlotSparse, labelled by the sites.
• Publishers' documents: CAWST's training material and the Save the Children Resource Centre's reports and guidance, each document labelled with the language its publisher catalogued.
• Not trained on: the Universal Declaration of Human Rights, kept for testing; passages quoting it were removed from training.

The training data is the full MillID build. The public baobabtech/millid-dataset is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its test split is the one scored above, without those passages.

Data processing

• Passages: Unicode NFKC, which folds the ligatures and presentation forms PDF text layers are full of; URLs, emails and dot leaders removed; a passage must be mostly letters. Exact duplicates were removed across every source, and no PDF gives more than 40 passages, article 20 or web page 10.
• Labels checked: a publisher's label is kept unless GlotLID is confident (≥ 0.8) the passage is in another major language, typically a copyright line; a passage from FinePDFs or FineWeb-2 is kept only when GlotLID's reading of the passage agrees with its document's.
• Text that is no language removed: passages GlotLID is confident are zxx (no linguistic content), mostly PDFs whose fonts map letters to the wrong code points, and documents catalogued in the wrong language.
• Splits by document: no test passage shares a document with a training one.

Training

1,944,095 passages; the best validation macro F1 at step 56,000, after 3.4 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.0003, cosine schedule, label smoothing 0.1.

Architecture

Drawn at the medium preset's sizes (12 blocks, d = 768); this model's are in the table above.

UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (32,768 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 2 bytes per token -> 12 pre-norm blocks (RMSNorm, attention with RoPE, 12 heads, SwiGLU; d=768) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. See ARCHITECTURE.md.

Usage and limitations

Intended usage

• Routing documents: choosing the OCR, translation or language model for a document by its language.
• Building and filtering corpora: sorting web or PDF text by language, low-resource languages included.
• Describing collections: which languages a set of documents holds, and how much of each.
• On-device detection: in a browser or on a phone with tiny v1.5, where text should not leave the device.

Limitations

• Close languages: languages that differ by a few words are its weakest (F1 on test: Croatian 0.69, Bosnian 0.67, Malay 0.75, Indonesian 0.78, Norwegian Bokmål 0.81). Their short passages are often genuinely ambiguous.
• Text that is no language: there is no "no language" class. Nepali typed in the legacy Preeti font, whose text layer is Latin letters standing for Devanagari, was labelled as another language by MillID models, some with a top probability up to 0.74.
• One label per passage: a passage that mixes languages gets one; identify_document gives the segments of a longer text.
• Labels learned from GlotLID: the FinePDFs and FineWeb-2 passages were labelled by GlotLID, so MillID learned its labels, mistakes included; udhr, labelled by translators, is the independent test.
• Genre: for many low-resource languages the training text is mostly religious web text or children's stories, and a model can learn the genre along with the language.

Ethical considerations and risks

• Errors fall on the smallest languages: filtering or routing text by language can drop or mishandle the text of speakers of low-resource and closely related languages, where MillID is least accurate. Check the per-language scores of the languages that matter before filtering with it.
• A language is not an identity: do not use a language label to infer a person's nationality, ethnicity or origin.
• Labels can be contested: whether two varieties are one language or two is political as well as linguistic. MillID's labels follow ISO 639-3 and its training data, not a judgement on that question.

Benefits

MillID medium v1 is within 0.018 macro F1 of GlotLID, a 1.7 GB fastText model, on the 1,621 languages of test, and within 0.009 on udhr. All three sizes are open under Apache-2.0, with the code that trained and evaluated them and the dataset, less the text that is not ours to share.

Licence

The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the dataset card), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).