tiny-rerank-ja-en-30m-distilled
A 36.7M-parameter English-Japanese sentence encoder (256-dim, CPU-friendly), distilled from BAAI/bge-m3 into sbintuitions/modernbert-ja-30m. This is the checkpoint behind the "71% of the teacher" number in the project write-up.
Companion model: raihan-js/tiny-rerank-ja-en-30m is a further Matryoshka fine-tune of this checkpoint (use it if you want to truncate to 128 or 64 dimensions, or need the ONNX files). It is slightly weaker at full dimension.
Results
Cross-lingual EN→JA retrieval, 500 English queries against 5,000 Japanese passages (OPUS-100 en-ja pairs, so the evaluation is synthetic), nDCG@10, single run, CPU:
| Model | EN-JA nDCG@10 | JA-JA nDCG@10 (JQaRA subset) | Index MB (5k passages) |
|---|---|---|---|
| BAAI/bge-m3 (teacher, 568M) | 0.674 | 0.942 | 19.5 |
| cl-nagoya/ruri-v3-30m | 0.542 | 0.960 | 4.9 |
| this model | 0.481 (71% of the teacher) | 0.218 | 4.9 |
| intfloat/multilingual-e5-small | 0.480 | 0.911 | 7.3 |
| sbintuitions/modernbert-ja-30m (untrained base) | 0.037 | 0.322 | 4.9 |
| tiny-rerank-ja-en-30m (Matryoshka fine-tune) | 0.416 | 0.155 | 4.9 |
Per-item numbers are in the repo's results/baselines.json.
Read this before using it:
- The public
ruri-v3-30mis the same size and scores higher on both sets. This model is a measured distillation recipe, not a state-of-the-art encoder. - It is weak on Japanese-to-Japanese retrieval (0.22). Use it for English→Japanese queries only.
- The evaluation is synthetic (translated sentence pairs, subsampled to 5,000 passages), so absolute nDCG is inflated; only the comparisons are meaningful.
- About 5% of the eval queries (27 of 500) were also in the 50k training pairs. Scoring only the 473 unseen queries gives 0.484 vs 0.481 overall, so it does not drive the result (
scripts/leakage_check.py). - One seed, one run.
Training
Teacher: bge-m3 embeddings. Student: modernbert-ja-30m plus a linear projector (256 → 1,024, used only in training) trained to match the teacher's embedding of the same sentence. 50,000 OPUS-100 en-ja pairs (both sides), 3 epochs, batch 64, lr 2e-5, cosine-similarity loss, seed 0.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("raihan-js/tiny-rerank-ja-en-30m-distilled")
queries = ["The weather is lovely today."]
passages = ["今日はとても良い天気です。", "彼はスタジアムまで車で行った。"]
q = model.encode(queries, normalize_embeddings=True)
p = model.encode(passages, normalize_embeddings=True)
print(model.similarity(q, p)) # the first passage should score higher
No query/passage prefixes are needed. Embeddings are 256-dimensional.
Limitations and licence
MIT. Training pairs come from OPUS-100 (a mix of parallel corpora with their own terms). Not evaluated beyond the synthetic sets above; not tested for bias, long documents or domain text.
Author: Raihan Sikder · Code and write-up: https://github.com/raihan-js/tiny-bilingual-retriever