opus-mt-eu-es-onnx
ONNX export of Helsinki-NLP/opus-mt-eu-es (Basque -> Spanish), for use with optimum.onnxruntime.ORTModelForSeq2SeqLM.
Licence
Apache-2.0, taken from the base model card (Helsinki-NLP/opus-mt-eu-es).
Export command
optimum-cli export onnx --model Helsinki-NLP/opus-mt-eu-es --task text2text-generation-with-past --no-post-process eu-es
int8 dynamic quantization was then applied to each ONNX file with onnxruntime.quantization.quantize_dynamic (QUInt8).
File layout
encoder_model.onnx
decoder_model.onnx
decoder_with_past_model.onnx
config.json / generation_config.json
source.spm / target.spm / vocab.json / tokenizer_config.json / special_tokens_map.json
int8/
encoder_model.onnx
decoder_model.onnx
decoder_with_past_model.onnx
Sanity check
3 short sentences, PyTorch original vs this ONNX export, num_beams=4, max_new_tokens=48:
| input | PyTorch | ONNX fp32 | ONNX int8 |
|---|---|---|---|
| Kaixo, zer moduz? | Hola, ¿cómo estás? | Hola, ¿cómo estás? | Hola, ¿cómo estás? |
| Txakurra parkean korrika ari da. | El perro corre en el parque. | El perro corre en el parque. | El perro corre por el parque. |
| Egun on guztioi. | Buenos días a todos. | Buenos días a todos. | Buenos días a todos. |
fp32 matches PyTorch 3/3. int8 output recorded above (not gated).
Usage
from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSeq2SeqLM
model_id = "TigreGotico/opus-mt-eu-es-onnx"
tok = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForSeq2SeqLM.from_pretrained(model_id)
inputs = tok("Kaixo, zer moduz?", return_tensors="pt")
out = model.generate(**inputs, num_beams=4, max_new_tokens=48)
print(tok.decode(out[0], skip_special_tokens=True))
For the int8 quantized version, load from the int8/ subfolder:
model = ORTModelForSeq2SeqLM.from_pretrained(model_id, subfolder="int8")