decosa-qe-xlmr-large (v2)

A translation quality-estimation (QE) model you can use commercially. Given an English source sentence and its translation into German, French, Spanish, Italian, Dutch or Polish, it returns:

• p_error: the probability that the translation changes the meaning of the source (a segment score; score = 1 - p_error);
• the likeliest error category: number, negation, omission, addition, entity, roles, hedge or term;
• word spans in the source and the translation tagged with a category (fluency is tagged but does not count as a meaning error).

It runs on a CPU: about 90-140 ms for one sentence pair and 40-60 ms per pair in batches of 32 (8 threads, AMD Ryzen 9 9950X3D shared with other services, fp32). It was built to pre-filter an LLM meaning check (a back-translation plus a 27B judge, about 7 s and $0.00026 per line), not to replace a person who reads the language.

Why it exists: the strong open QE models (Unbabel CometKiwi, XCOMET) are CC BY-NC-SA, so they cannot be used commercially. This model starts from an MIT encoder and is trained only on data whose licence allows commercial use.

Results

Held-out test sets (never trained on); thresholds set on dev data only. "LLM" is a back-translation plus Qwen3.8-27B meaning check, the same pairs judged by it.

Per error type on the planted meaning set: dropped clause 143/144, hedge 83/90, negation 88/106, entity 103/132, roles 59/90. Word-level error spans (F1, any category): 0.86 on number drift, 0.53 on the meaning set.

Intended use

• Screening machine translations of short factual text (product safety, patient leaflets, lay summaries of trials, notices) for changed meaning: a flipped negation, a dropped condition, a changed number, a swapped name or role.
• As the first stage of a cascade: lines it is sure about are passed or flagged; the uncertain rest go to a slower checker or a person.

Not for: judging fluency or style, grading human translators, languages other than the seven listed, source languages other than English, long documents (it reads at most 256 sub-word tokens per pair; split into sentences), or any decision about a person without human review. A low p_error means no error of the trained kinds was found; it does not mean the translation is right.

Limitations (measured, see Results)

• Word-sense mistranslations (a clinical "trial" rendered as a court case) are not caught: 0 of 9 real ones.
• Plausible swaps of people or things ("the seller" -> "the buyer", "safety glasses" -> "gloves") and negations that move within the sentence are missed more often than by the LLM check.
• Written-out abbreviations and decimal commas in clinical text trigger false flags (18 of 21 on the real set); literary text gets about 9% false flags.
• Trained and evaluated on synthetic planted errors; the real-error sample is small (15 errors) and was labelled by the same team that built the data, without a native-speaker review.
• English source only; German, French, Spanish, Italian, Dutch and Polish targets only.

Training data (every source and its licence)

All training pairs are synthetic: a correct translation, or the same translation with one planted error. Nothing was labelled by people.

Not used: FLORES-200 (CC-BY-SA 4.0; used only for evaluation), MLQE-PE (repository CC0, but its sources are Wikipedia text and only English-German is relevant), WMT QE shared-task data (licences vary or are unclear), CometKiwi / XCOMET weights and outputs (CC BY-NC-SA), any NC checkpoint.

How errors were planted: rule-based minimal edits to the English (negation flipped, hedge changed, clause or qualifier dropped, words added, a named thing or a term swapped, two people swapped, a number changed) were machine translated and paired with the original English, so the wrong translation is fluent; target-side plants (a digit, a unit, a date, a negation removed, a word-sense error, word order or a doubled word for fluency) were made directly in clean translations. Sentences that overlap the evaluation sets were removed before training. The generator and the held-out test sets are not released.

Model

XLM-RoBERTa-large (FacebookAI/xlm-roberta-large, rev c23d21b0, MIT) reads source </s></s> translation. Heads: one logit for a meaning error (mean + first-token pooling), a softmax over ten segment categories, and a per-token softmax over OK and nine categories. Trained on an Apple M3 Ultra (MPS), AdamW, bf16 autocast. v1: 3,000 steps from the base on 72,034 pairs (lr 1.5e-5); v2: warm start, 2,113 steps on 100,478 pairs (lr 8e-6), checkpoint chosen on the dev split (step 700). About 3 h in total.

Files: model.safetensors (fp32), config.json (the encoder), tokenizer files, qe_config.json (categories, the calibrated thresholds, max length), SHA256SUMS.

Usage

import json, torch
from safetensors.torch import load_file
from transformers import AutoConfig, AutoTokenizer
from qe_model import QEModel, CATS, TAGS          # qe_model.py ships with the weights

d = "decosa-qe-xlmr-large"
tok = AutoTokenizer.from_pretrained(d)
m = QEModel(config=AutoConfig.from_pretrained(d)); m.load_state_dict(load_file(f"{d}/model.safetensors")); m.eval()
th = json.load(open(f"{d}/qe_config.json"))["thresholds"]

src = "Do not give this medicine to children under 3 years."
mt = "Geben Sie dieses Arzneimittel Kindern unter 3 Jahren."
enc = tok(src, mt, truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    seg, cat, toks = m(enc["input_ids"], enc["attention_mask"])
p = torch.sigmoid(seg).item()
print(round(p, 3), "error" if p >= th["error"] else "check" if p >= th["check"] else "ok", CATS[int(cat.argmax())])

Licence

Apache-2.0 for the weights and code in this repository. The base model is MIT (XLM-RoBERTa, Meta AI). The training data sources above ask for attribution; it is given here and in NOTICE.

Citation and contact

Decosa, decosa.ai. Evaluation: docs/evals/qe-model.md in the Decosa API repository (numbers reproduced above).