Small Transformer (≈11.5M params, d_model=256, 4 encoder/decoder layers) trained from scratch for Spanish→Quechua (Chanka) translation, using PRPE (Prefix-Root-Postfix Encoding) — a morphologically-guided tokenizer — on the target (Quechua) side, 22k-piece vocabulary (98.3% token coverage on train). Compared against an identical model trained with standard BPE (quechua-chanka-mt-bpe) in a controlled ablation isolating the effect of the tokenizer.
Introduced as the extrinsic evaluation in QuechuaTok (Contreras, 2026), which shows that PRPE's morphological boundary accuracy (MorphAcc) — not just intrinsic fertility/perplexity — translates into real translation quality gains.
This model's target-side tokenizer (PRPE) is a custom morphological segmenter,
not a standard SentencePiece/BPE tokenizer, so it is not compatible with the
Hugging Face Inference widget or AutoTokenizer. Use the manual snippet
below instead. (The BPE sibling model, linked above, does support the
standard widget.)
| Tokenizer | MorphAcc% | BLEU | chrF |
|---|---|---|---|
| BPE | 6.67 | 4.18 | 35.13 |
| PRPE (this model) | 83.33 | 7.05 | 41.10 |
+68.7% BLEU / +17.0% chrF over the BPE baseline, despite longer target sequences (18.6 vs 11.4 tokens/sentence) — morphological boundaries give the model more learnable, compositional units than memorizing surface forms.
from huggingface_hub import hf_hub_download
from transformers import MarianMTModel
import sentencepiece as spm
import sys, json, torch
repo_id = "makitty/quechua-chanka-mt-prpe"
# source tokenizer (Spanish, standard SentencePiece)
src_spm_path = hf_hub_download(repo_id, "source.spm")
sp_src = spm.SentencePieceProcessor()
sp_src.load(src_spm_path)
# target tokenizer (PRPE, custom morphological segmenter)
tok_module_path = hf_hub_download(repo_id, "tokenizer_prpe.py")
sys.path.insert(0, tok_module_path.rsplit("/", 1)[0])
import tokenizer_prpe as prpe # provides tokenize_prpe_sentence / detokenize_prpe
vocab_path = hf_hub_download(repo_id, "prpe_vocab.json")
vocab = json.load(open(vocab_path, encoding="utf-8"))
inv_vocab = {v: k for k, v in vocab.items()}
model = MarianMTModel.from_pretrained(repo_id)
def translate(text):
ids = sp_src.encode(prpe.normalize(text), out_type=int)[:62] + [sp_src.eos_id()]
input_ids = torch.tensor([ids])
out = model.generate(input_ids=input_ids, max_length=64, num_beams=4, decoder_start_token_id=1)
toks = [inv_vocab.get(i, "") for i in out[0].tolist() if i not in (0, 1, 2)]
return prpe.detokenize_prpe(toks)
print(translate("hola como estas"))
makitty/quechua-chanka-unified-v2,
broad_train split (91k parallel sentence pairs).
@misc{contreras2026quechuatok,
title={QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric
for Tokenizer Evaluation in Agglutinative Low-Resource Languages},
author={Contreras, Maria},
year={2026},
eprint={2606.23943},
archivePrefix={arXiv}
}
model.safetensors 69 MBtraining_args.bin 5.3 kB