rikai-translate/rikai-jp-eng-Bench-265-1.4B

🔥 The Bench 265 models are tiny, powerful bidirectional translation models for translating Japanese and English. 🔥

Models

Evaluation at a glance

rikai 1.4B matches/beats CAT-Translate-3.3B on most automatic metrics at 2.4× its decode speed. This model acheived the best scores across all metrics relative to its size.

Evaluation

1. 26-domain gate (human references)

This is our main test battery: 1,550 human-referenced items in 26 sets.

How it was run:

• None of these items were used in training. All training data was screened against them, using 12-character shingles.
• Every model is greedy-decoded at 8-bit (MLX) with its own recommended prompt. Ours is decoded with the prefix mask.

2. chrF++ and BLEU

Per-domain results for every metric →

3. LLM as Judge results

The only thing benchmaxing here is me, so this model was also trained with an LLM judge.

Judge: Claude Sonnet 5.5, on 500 blind items it never saw in training (350 ja→en, 150 en→ja).

Prompt: it sees the source, a reference marked as "one acceptable translation", and one model's translation, never two side by side. It scores it 0–100 (100 perfect, 75 minor errors, 50 a major error, 25 meaning wrong, 0 unusable) and lists any errors as minor, major or critical. Every item is judged twice, in differently shuffled batches, and the two scores are averaged.

Larger models score higher and are left off the chart: CAT-3.3B-β 70.4, CAT-3.3B 71.6, CAT-7B 73.4, Tower-Plus-9B 74.0, TranslateGemma-12B 75.5.

Usage

The prompt and chat template are the same as CAT-Translate's:

Translate the following {src_lang} text into {tgt_lang}.

{src_text}

It runs on a custom prefix mask, so the prompt gets read in both directions before the first word comes out. Here's an MLX example (Apple silicon, 8-bit build):

import mlx.core as mx, mlx_lm.models.llama as llama
from mlx_lm import load
from mlx_lm.models.cache import make_prompt_cache
from huggingface_hub import snapshot_download

path = snapshot_download("rikai-translate/rikai-jp-eng-Bench-265-1.4B", allow_patterns="mlx-8bit/*") + "/mlx-8bit"
model, tok = load(path)
causal_mask = llama.create_attention_mask

def translate(text, src="Japanese", tgt="English", max_tokens=1024):
    msg = [{"role": "user", "content": f"Translate the following {src} text into {tgt}.\n\n{text}"}]
    ids = tok.apply_chat_template(msg, add_generation_prompt=True)
    cache = make_prompt_cache(model)
    llama.create_attention_mask = lambda *a, **k: None   # prompt[:-1] attends in both directions
    model(mx.array([ids[:-1]]), cache=cache)
    llama.create_attention_mask = causal_mask            # generation stays causal
    logits, out = model(mx.array([ids[-1:]]), cache=cache), []
    for _ in range(max_tokens):
        t = mx.argmax(logits[0, -1]).item()
        if t in tok.eos_token_ids: break
        out.append(t); logits = model(mx.array([[t]]), cache=cache)
    return tok.decode(out).strip()

print(translate("駅前の新しいパン屋、朝七時から開いてるらしいよ。"))
# The new bakery in front of the station seems to be open from seven in the morning.

License

MIT.

Citation

If you use this model, please cite the base model:

@misc{jinnai2026cattranslatebuildingcompactopensource,
      title={CAT-Translate: Building Compact Open-Source Models for Japanese-English Translation},
      author={Yuu Jinnai},
      year={2026},
      eprint={2606.21413},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.21413},
}

Acknowledgments

• CAT-Translate by CyberAgent: the base model.
• Hy-MT2-7B by Tencent: the teacher.
• sarashina2.2 by SB Intuitions: the pre-trained base under CAT-Translate.
• Metrics: MetricX-24 (Juraska et al.), COMET-22 and CometKiwi (Rei et al.), and sacrebleu (Post).
• Claude Sonnet 5.5 by Anthropic: the LLM judge.
• Data: FLORES-200, WMT23 / WMT24++, NTREX-128, EJMMT, OpenMantra, Tatoeba, BSD, JESC and pfmt-bench-fin-ja.
• MLX and mlx-lm by Apple: training and inference.

About

current Bench PR 265

_{Bayla, head of QA.}