MADLAD-400-3B-MT for Asturian, Aragonese and Aranese

google/madlad400-3b-mt fine-tuned to translate into and out of three Romance languages of the Iberian Peninsula: Asturian (asturianu), Aragonese (aragonés) and Aranese (aranés, the Occitan of the Val d'Aran). One model, one target tag per language:

• Into the three languages from Spanish, Catalan, English and French, and out of them into the same four languages with the base model's tags (<2es>, <2ca>, <2en>, <2fr>). Other source languages of the base model may work, but they are not tested.
• <2es> still gives Spanish and <2oc> standard (Languedocien) Occitan, and the other languages keep working as in the base model, with a small loss (see Other languages).
• Same size and speed as the base model (3B parameters, T5 architecture). A CTranslate2 int8 version is available at Ugiat/madlad400-3b-mt-ast-an-oc_aranes-ct2.
• How it compares with the other models for these languages: Evaluation.

The model is dedicated to the TAN-IBE project (Neural machine translation for the Romance languages of the Iberian Peninsula, Universitat Oberta de Catalunya), whose openly published corpora made the Asturian and Aragonese parts of this model possible. It is not a TAN-IBE model: it was prepared by Ugiat Technologies as an independent work, and it is not produced, reviewed or endorsed by the TAN-IBE project, by Google or by the MADLAD-400 authors.

Why this model

MADLAD-400-3B-MT has no Asturian tag, its <2an> often answers in Spanish or produces unrelated text, and its <2oc> produces standard Occitan, not Aranese. (Its <2arn> tag is Mapudungun, ISO 639-3 arn, not Aranese.)

More examples, into each language (this model, English source):

These examples were not reviewed by native speakers. See Limitations for the kinds of error we have seen.

Usage

transformers

from transformers import AutoTokenizer, T5ForConditionalGeneration

repo = "Ugiat/madlad400-3b-mt-ast-an-oc_aranes"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = T5ForConditionalGeneration.from_pretrained(repo, dtype="auto", device_map="auto")

def translate(text, tag):
    ids = tokenizer(f"{tag} {text}", return_tensors="pt").input_ids.to(model.device)
    out = model.generate(input_ids=ids, max_new_tokens=128, num_beams=4)
    return tokenizer.decode(out[0], skip_special_tokens=True)

for tag in ("<2ast>", "<2an>", "<2oc_aranes>"):
    print(tag, translate("Tomorrow we will go to the market to buy fruit and vegetables.", tag))
print(translate("Mañá vamos dir al mercáu a mercar fruta y verdura.", "<2en>"))   # out of Asturian

• Use transformers 4.x (tested with 4.57). transformers 5.x does not load MADLAD-400 checkpoints correctly and produces garbage, for this model and for the original one.
• Use this repository's tokenizer: <2oc_aranes> (id 256000) and <2ast> (id 256001) are SentencePiece pieces, defined like MADLAD-400's own <2xx> tags. The base model's tokenizer does not know them.
• As with the base model, only the target language is tagged: the source needs no tag.

CTranslate2 (faster, int8)

import ctranslate2
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

path = snapshot_download("Ugiat/madlad400-3b-mt-ast-an-oc_aranes-ct2")
tokenizer = AutoTokenizer.from_pretrained(path)
translator = ctranslate2.Translator(path, device="cuda", compute_type="int8_float16")   # or device="cpu", compute_type="int8"

def translate(sentences, tag):
    batch = [tokenizer.convert_ids_to_tokens(tokenizer.encode(f"{tag} {s}")) for s in sentences]
    return [tokenizer.decode(tokenizer.convert_tokens_to_ids(r.hypotheses[0]), skip_special_tokens=True)
            for r in translator.translate_batch(batch, beam_size=4)]

print(translate(["El museo estará cerrado durante las obras de reforma."], "<2an>"))

Evaluation

FLORES+ devtest (1,012 sentences), BLEU / chrF (higher is better), sacrebleu 2.6 defaults. We ran every model ourselves on the same test files, with the decoding of their model cards (notes below the tables). A dash means the model does not translate that direction.

Models compared: TAN-IBE Marian models (aoliverg/MTUOC-Marian-TAN-IBE-*, Spanish ↔ each language, under 0.1B parameters, GPL-3.0), SalamandraTA-7B (GPL-3.0) and 2B (Apache-2.0), IbRo-nllb (1.3B, Apache-2.0) and the base model.

Asturian

Spanish ↔ Asturian scores are low for every model: the FLORES+ Asturian sentences are free translations, far from the Spanish sentence structure.

Aragonese

‡ Spanish → Aragonese: a difference of written variety. The FLORES+ Aragonese reference writes the articles lo, la, los, las and per, pa, puede; Aragonese Wikipedia, where all of this model's human Aragonese comes from, mostly writes o, a, os, as and por, ta, puet, and so does this model. Word counts over the 1,012 Spanish → Aragonese outputs:

TAN-IBE and IbRo-nllb, trained largely on Apertium output, write like the reference. On 224 held-out Spanish–Aragonese sentence pairs from Wikipedia (TAN-IBE's corpus), which none of our models was trained on, the order reverses: this model 50.9 / 73.3, TAN-IBE 45.0 / 70.2, IbRo-nllb 42.6 / 69.4. That test is in this model's training domain and may be in TAN-IBE's training data, so neither test settles which model is better; each favours its own variety.

Aranese

Held-out test of 1,000 Catalan–Aranese pairs from the BSC corpus (in this model's training domain; the corpus' Catalan side was partly generated by SalamandraTA-7B): Catalan → Aranese 51.9 / 73.3 (SalamandraTA-7B 46.7 / 70.3, IbRo-nllb 45.5 / 69.6), Aranese → Catalan 61.3 / 77.5 (SalamandraTA-7B 65.1 / 80.7).

Aranese, not standard Occitan. 79–82 % of this model's FLORES+ outputs into Aranese contain typically Aranese words (eth, deth, ath, ena, damb, ua…) and 1–3 % typically Languedocien words (lo, los, dins, amb, pas…); the reference: 80 % and 1 %; the base model's <2oc>: 2–4 % and 87–91 %.

Notes on the evaluation

• How we ran the other models. TAN-IBE: converted to CTranslate2, beam 12 (their configuration). SalamandraTA: its chat template and prompt (Translate the following text from {source} into {target}.), beam 5. IbRo-nllb and aina-translator-es-oc: beam 5, codes ast_Latn, arg_Latn, arn_Latn. This model: beam 4.
• The Aranese FLORES+ reference was made by translating the Spanish sentences with Apertium, then post-editing and revising them (PILAR), so Spanish → Aranese scores are much higher than English → Aranese scores for every model.
• † IbRo-nllb into Catalan mostly answers in Aranese instead of Catalan.
• The aina-translator-es-oc card reports about 30 BLEU: it was evaluated before a 2025 fix to the Aranese FLORES+ devtest, about half of which was misaligned (PILAR README). The table gives its score on the corrected devtest.
• Right language. Share of outputs with at least two typically Asturian words: 61–64 % for this model, 58 % for the FLORES+ reference, 57–58 % for SalamandraTA-7B. Aragonese (two typically Aragonese words, few Spanish-only words): 69–74 % for this model, 59 % for the reference, 57–61 % for SalamandraTA-7B. Outputs into Spanish look Asturian or Aragonese in 1 % of cases, as often as the Spanish reference does.
• None of the FLORES+ dev or devtest sentences is in this model's training data. We checked for exact matches and for any shared 13-word sequence. All scores above are on devtest.

Other languages (FLORES-200 devtest, forgetting check)

Every direction loses a little, −0.69 chrF++ on average, most for Chinese and Japanese. English → Spanish: chrF++ 54.1 → 53.5, spBLEU 32.9 → 31.9.

Training

LoRA fine-tune of the base model with two new tag embeddings (<2oc_aranes> initialised from <2oc>, <2ast> from <2es>), merged into the weights. One epoch over 1.61M pairs plus 0.53M replay pairs (the base model's own translations of MADLAD-400 sentences, to limit forgetting).

• Human text on the target side for the three languages. Every Asturian, Aragonese and Aranese target is human-written; machine translations (the base model's, TAN-IBE's Marian and Apertium translations) are used as sources. Machine-translated targets appear only out of these languages (the base model's translations into Spanish, Catalan, English and French), in the contrastive pairs below and in the replay pairs.
• Contrastive pairs. The same source is paired with <2ast> / <2an> → the human sentence and with <2es> → its Spanish, and with <2oc_aranes> → Aranese and <2oc> → standard Occitan, so that each tag keeps its own language.
• Asturian and Aragonese: the TAN-IBE Wikipedia corpora (parallel Spanish–Asturian and Spanish–Aragonese sentences, and monolingual Wikipedia filtered with marker words), back-translated by the base model from Spanish, Catalan, English and French; Aragonese Wikipedia with TAN-IBE's Apertium Spanish as source; Apertium translations of human Spanish (TAN-IBE's NTEU files, 40,000 sentences per language) as sources into Spanish only.
• Aranese: the corpora listed under License and attribution, back-translated, plus TAN-IBE's Aranese Wikipedia and NTEU files.

Limitations

• SalamandraTA-7B scores higher on FLORES+ almost everywhere. It is more than twice the size of this model and is GPL-3.0.
• Spanish → Aragonese follows Aragonese Wikipedia's usage (o, a, os, as, ta, por), not the FLORES+ reference's (lo, la, los, las, pa, per). Both are written Aragonese, but if you need the latter, TAN-IBE and IbRo-nllb match it much better.
• Catalan, English and French → Aragonese are weak for every model, this one included (15.8–19.6 BLEU).
• Short, informal sentences sometimes come out in Spanish with <2ast> (e.g. Oye chicos, ¿vienen a la fiesta el sábado?), and meaning errors occur, such as a wrong person (nun pudo llamar-y, "he couldn't call him", for "I couldn't call you") or a wrong word (Aragonese charrar, "to talk", for "to call").
• Do not use <2es> on Spanish input. The tag was trained on other source languages; on Spanish input it can answer in Asturian.
• Into Aranese, our Aranese-only model Ugiat/madlad400-3b-mt-oc_aranes scores slightly higher (0.4–1.2 BLEU) on FLORES+.
• Spelling and grammar are learned from the data, mostly Wikipedia for Asturian and Aragonese. The Aragonese data follows the norm of the Academia Aragonesa de la Lengua; the output may not always follow the norms of the Academia de la Llingua Asturiana, the Academia Aragonesa de la Lengua or the Institut d'Estudis Aranesi.
• Automatic evaluation only, against single references. The outputs were not reviewed by native speakers.
• Other languages lose a little quality (−0.69 chrF++ on average over 13 FLORES-200 directions). If you do not need these languages, the base model is slightly better.
• Base model limitations apply: MADLAD-400-3B-MT can hallucinate, omit content or mistranslate, especially on long, noisy or low-resource inputs.

License and attribution

• License: Apache-2.0, the license of the base model google/madlad400-3b-mt.
• Training data includes (used for training only; this repository contains no training text):
  • TAN-IBE corpora (Zenodo), Antoni Oliver, Universitat Oberta de Catalunya: Asturian (10.5281/zenodo.17611469, CC BY 4.0), Aragonese (10.5281/zenodo.17584219, CC BY-SA 4.0) and Aranese (10.5281/zenodo.17611857, CC BY 4.0). They are built from Wikipedia (CC BY-SA 4.0) and from Apertium translations of the Spanish side of the NTEU corpus, all under the records' licences.
  • BSC-LT/Catalan-Aranese_Parallel_Corpus, CC BY 4.0, Language Technologies Unit of the Barcelona Supercomputing Center and its sources.
  • PILAR, Transducens, Universitat d'Alacant (packaging CC0; the texts belong to their sources).
  • projecte-aina/ES-OC_Parallel_Corpus, CC BY-SA 4.0, Projecte Aina.
  • MADLAD-400, ODC-BY 1.0, built from Common Crawl.
  • Occitan Wikipedia (CC BY-SA 4.0) and Mozilla Common Voice (CC0).
• Evaluation data: FLORES+ (CC BY-SA 4.0), evaluation only.

Please cite the MADLAD-400 paper, which introduced the base model, and PILAR:

@misc{kudugunta2023madlad400,
      title={MADLAD-400: A Multilingual And Document-Level Large Audited Dataset},
      author={Sneha Kudugunta and Isaac Caswell and Biao Zhang and Xavier Garcia and Christopher A. Choquette-Choo and Katherine Lee and Derrick Xin and Aditya Kusupati and Romi Stella and Ankur Bapna and Orhan Firat},
      year={2023},
      eprint={2309.04662},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{PILAR,
      editor={Galiano-Jiménez, Aarón and Sánchez-Martínez, Felipe and Pérez-Ortiz, Juan Antonio},
      title={PILAR},
      url={https://github.com/transducens/PILAR},
      year={2024}
}

Suggested reference for this model:

@misc{madlad400_3b_mt_ast_an_oc_aranes,
      title={MADLAD-400-3B-MT for Asturian, Aragonese and Aranese},
      author={Ugiat},
      organization={Ugiat Technologies},
      year={2026},
      howpublished={\url{https://huggingface.co/Ugiat/madlad400-3b-mt-ast-an-oc_aranes}}
}

Contact

Ugiat Technologies, ugiat@ugiat.com.