ModernMT-en-ru-EXP
Экспериментальная модель для перевода с английского на русский на базе deepvk/RuModernBert-small. Модель была инициализирована методом Bert2Bert, подробнее описанным в этой статье.
An experimental model for translating from English to Russian based on deepvk/RuModernBert-small. The model was initialized using the Bert2Bert method, described in more detail in this article.
Usage
from transformers import AutoModel, AutoTokenizer
model_name = "PruhaNLP/ModernMT-en-ru-EXP"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
model.to("cuda").eval()
text = "The quick brown fox jumps over the lazy dog."
inputs = tokenizer(text, return_tensors="pt").to("cuda")
output_ids = model.generate(
inputs["input_ids"],
attention_mask=inputs["attention_mask"],
max_length=256,
num_beams=4,
)
translation = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(translation)
Evaluation
| Model | Params | FLORES-200 | WMT13 | WMT14 | WMT15 | WMT16 | WMT17 | WMT18 | WMT19 | WMT20 | WMT21 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| facebook/wmt19-en-ru | ~300M | 30.4 | 29.7 | 43.1 | 40.3 | 35.8 | 42.2 | 34.9 | 33.4 | 23.8 | — |
| PruhaNLP/ModernMT-en-ru-EXP | 66M | 29.5 | 24.8 | 38.9 | 32.0 | 30.1 | 33.9 | 29.9 | 29.8 | 23.2 | 25.3 |
| facebook/nllb-200-3.3B | 3.3B | 29.3 | 27.4 | 39.8 | 33.2 | 32.6 | 34.9 | 31.3 | 32.0 | 23.6 | 37.5 |
| facebook/nllb-200-distilled-1.3B | 1.3B | 28.5 | 27.4 | 39.5 | 33.5 | 32.8 | 34.8 | 31.7 | 32.2 | 23.6 | 37.3 |
| facebook/nllb-200-1.3B | 1.3B | 28.3 | 26.7 | 38.5 | 33.1 | 32.0 | 34.3 | 30.6 | 31.6 | 23.4 | 36.5 |
| facebook/m2m100_1.2B | 1.2B | 28.1 | 24.3 | 37.0 | 30.5 | 28.9 | 32.5 | 28.1 | 28.2 | 22.7 | — |
| gsarti/opus-mt-tc-base-en-ru | ~76M | 27.6 | 23.4 | 34.7 | 29.0 | 27.5 | 30.6 | 27.1 | 26.8 | 20.8 | — |
| facebook/nllb-200-distilled-600M | 600M | 25.6 | 25.0 | 35.4 | 29.9 | 29.1 | 31.4 | 27.8 | 29.1 | 21.6 | 32.7 |
| facebook/m2m100_418M | 418M | 22.5 | 20.5 | 30.4 | 25.6 | 24.0 | 26.4 | 22.7 | 23.4 | 18.6 | — |
Training
Энкодер был целиком инициализирован RuModernBert-small, декодер - каждым вторым слоем энкодера. lr трапециевидный - 5% warmup, 20% decay. В конце были смерджены последние 7 чекпоинтов.
The encoder was initialized entirely by RuModernBert-small, and the decoder was initialized by every second layer of the encoder. Trapezoidal lr - 5% warmup, 20% decay. At the end, the last 7 checkpoints were merged.
Data
Модель была обучена на 100млн случайных пар из датасета Helsinki-NLP/tatoeba_mt_train без какой либо умной фильтрации - только по числу символов дабы исключить обрезанные text.
The model was trained on 100 million random pairs from the Helsinki-NLP/tatoeba_mt_train dataset without any clever filtering - only by the number of characters in order to exclude cropped text.
Hardware
Одна V100)))
Single V100
Лицензия
Apache 2.0