Multilingual Educational-Quality Classifier (small, 141M)

Scores how educational a web page is, from 0 (none) to 5 (excellent, textbook-like), in English and 65 other languages, on the same scale as FineWeb-Edu. Use it to filter or weight pretraining data, for example FineWeb-2, the way FineWeb-Edu filtered English (int_score >= 3). Built on mmBERT-small, Apache-2.0, with ONNX files included. A larger, more accurate version is available as Horizon-Labs/multilingual-edu-classifier-base.

• Drop-in for code written for HuggingFaceFW/fineweb-edu-classifier: one regression output; int_score = round(clamp(score, 0, 5)).
• It reads the first 512 tokens of a document. onnx/model_quantized.onnx (int8 embeddings, 268 MB): scores differ from fp32 by 0.004 on average; the educational / not decision (2.5) agrees for 100.0% of 400 test documents.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("Horizon-Labs/multilingual-edu-classifier-small")
model = AutoModelForSequenceClassification.from_pretrained("Horizon-Labs/multilingual-edu-classifier-small")
texts = ["La photosynthèse est le processus par lequel les plantes ...", "Buy cheap shoes now! Free shipping ..."]
inputs = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=512)
with torch.no_grad():
    score = model(**inputs).logits.squeeze(-1).float()
int_score = score.clamp(0, 5).round().int()
print(score.tolist(), int_score.tolist())

Evaluation

• 65 languages: 9,750 held-out FineWeb-2 documents (150 per language; never used for training) scored with the FineWeb-Edu rubric by an independent judge, google/gemma-4-31b-it (a different model family from our teacher). Metric: Spearman rank correlation with the judge, over all documents and averaged per language.
• English: 6,000 documents from a held-out shard of HuggingFaceFW/fineweb-edu-llama3-annotations (Llama-3-70B scores); Spearman, and F1 of the educational decision (score >= 3, as FineWeb-Edu uses it).

• The table shows the released checkpoint. Means over 2 training seeds (small: 65 languages .823 / .823, English .781 / .780; base: .847 / .845, .793 / .787).
• The English classifiers rank multilingual documents poorly (Spearman about 0.3); this model is built for that case. On English, this model matches HuggingFaceFW/fineweb-edu-classifier (Spearman 0.781 vs 0.781). HuggingFaceFW/fineweb-edu-classifier was trained on the same Llama-3 annotations as the English test set (in-domain for it).
• Caveat: the multilingual reference is itself an LLM. The judge agrees with Llama-3 about as well as our teacher does (English Spearman 0.73 vs 0.70), but the judge and our teacher agree with each other more (0.88), which favours this model somewhat on that test. No human-labelled multilingual educational-quality test set was available.
• Per language (Spearman with the judge): highest hun_Latn 0.88, cmn_Hani 0.88, eus_Latn 0.88, nob_Latn 0.88, swe_Latn 0.87, tur_Latn 0.87, lit_Latn 0.87, cat_Latn 0.87; lowest amh_Ethi 0.65, sin_Sinh 0.68, hye_Armn 0.68, urd_Arab 0.69, kat_Geor 0.70, mal_Mlym 0.70, npi_Deva 0.71, mya_Mymr 0.71.

Training

• English: 149,997 documents with their Llama-3-70B scores from HuggingFaceFW/fineweb-edu-llama3-annotations (ODC-BY).
• 65 other languages: 129,965 documents from FineWeb-2 (ODC-BY), about 2,000 per language, scored by Qwen3.8-27B (Apache-2.0) with the same rubric (expected score from its answer probabilities). Those scores were mapped onto the Llama-3 scale by isotonic regression fitted on 10,000 English documents scored by both, so that 3 means the same thing in every language.
• mmBERT-small with one regression output, mean squared error, max length 512 tokens; checkpoint chosen by Spearman on 13,920 held-out documents. Code: code/ in this repository.

Limitations

• "Educational" here means the FineWeb-Edu rubric (useful for teaching at primary to grade-school level). It undervalues good content that is not school-like (fiction, news, forums, specialised research), and it can encode the judges' cultural biases. Filtering with it changes what a model trained on the data sees, so check that the result still covers what you need.
• Labels come from LLMs, not people, and the per-language agreement varies (see above). Lower-resource languages and scripts are weaker.
• Only the first 512 tokens are read.