Bangla-Embed-E5-Banglish

A compact 118M Bengali sentence encoder that additionally supports romanized Bengali (Banglish) and, uniquely, retrieves across scripts — aligning romanized queries with their native-Bengali and English counterparts. Rebased from intfloat/multilingual-e5-small by three-stage distillation from a BGE-M3 teacher, with IndicXlit-romanized views added in the distillation and contrastive stages.

Highlights

Cross-script retrieval: on a held-out, human-typed romanization set (disjoint from the training transliterator) it retrieves native counterparts at acc@1 0.85 / 0.96, versus ≤0.27 for all baselines, and is the only model that retrieves across script in a controlled product-search study (MRR@10 0.31 vs ≤0.11).
MTEB(Indic) Bengali mean 0.698 — statistically tied with the 4.8× larger BGE-M3 (0.700) and mE5-large (0.690); strongest encoder in the ≤120M tier.

For native-Bengali-only use, the companion kazalbrur/bangla-embed-e5-small is slightly stronger on some full-corpus retrieval sets.

Comparison — MTEB(Indic) Bengali subset

Main score per task type (mteb 2.12), single shared harness. This model is bolded.

Model	Params	Retr	Class	Bitext-G	Bitext-C	Rerank	STS	Clust	Mean
BGE-M3 (teacher)	568M	0.644	0.879	0.874	0.722	0.852	0.593	0.340	0.700
bangla-embed-e5-small-banglish (this model)	118M	0.791	0.832	0.826	0.654	0.840	0.596	0.349	0.698
mE5-large	560M	0.631	0.847	0.876	0.748	0.852	0.540	0.339	0.690
bangla-embed-e5-small	118M	0.572	0.848	0.832	0.668	0.840	0.554	0.349	0.666
mE5-small (base)	118M	0.535	0.832	0.848	0.699	0.835	0.538	0.310	0.656
LaBSE	109M	0.442	0.804	0.849	0.705	0.792	0.583	0.239	0.631
Vyakyarth	300M	0.629	0.762	0.853	0.576	0.767	0.423	0.343	0.622
pm-mpnet-base	278M	0.337	0.749	0.618	0.426	0.701	0.355	0.370	0.508

Retr=BelebeleRetrieval, Class=BengaliSentiment, Bitext-G/C=IN22 Gen/Conv, Rerank=WikipediaReranking, STS=IndicCrosslingualSTS, Clust=SIB200. This model ranks second on mean and tops cross-lingual STS; the top-3 means span only 0.010 and are statistically indistinguishable under a 7-task paired bootstrap (treat sub-0.02 gaps as ties), i.e. on par with the 4.8× larger BGE-M3.

Cross-script (Banglish) retrieval — unique to this model

acc@1 on human-typed, held-out romanized queries retrieving native counterparts (disjoint from the training transliterator):

Model	bnₗₐₜ→en	bnₗₐₜ→bn
bangla-embed-e5-small-banglish (this model)	0.85	0.96
BGE-M3 / mE5-large / bangla-embed-e5-small	≤0.27	≤0.27

Only this model aligns romanized Bengali with native script/English.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("kazalbrur/bangla-embed-e5-small-banglish")
# romanized (Banglish) query -> native Bengali passage
q = model.encode(["dhaka kothay?"], prompt_name="query", normalize_embeddings=True)
d = model.encode(["ঢাকা বাংলাদেশের রাজধানী।"], prompt_name="passage", normalize_embeddings=True)
print((q @ d.T)[0, 0])

E5-family conventions: prefix queries with query: and passages with passage: (handled by prompt_name). Output dimension is 1024, L2-normalized.

Training & data

Three-stage curriculum (AdamW, cosine schedule, bf16). Distillation over ~18.7M EN–BN parallel pairs plus ~1M IndicXlit-romanized views; supervised contrastive fine-tuning (MNR) on ~2.77M pairs (Bangla-native core + SWIM-IR + MS MARCO-bn + ~0.32M romanized views); NLI polish (XNLI-bn, IndicXNLI-bn, MNLI-en). Released under the MIT license. Note that some training sources carry their own (in some cases non-commercial) terms — verify upstream data terms before commercial deployment.

Limitations

Training romanization is synthetic (IndicXlit); generalization is validated on a human-typed held-out set and a rule-based generator, but coverage is Standard Bangla only (regional typing conventions and varieties such as Sylheti/Chittagonian are unrepresented). On full-corpus MIRACL-bn/Mr.TyDi-bn it trails larger models and the untuned backbone; all metrics are automatic.

加载中…