bge-m3-hinglish-semcache
BGE-M3 fine-tuned so that English, romanised Hinglish and Hindi versions of the same question land close together, and near-miss questions (same words, different answer) land further away. Trained on the train splits of a customer-support benchmark (Patle, Dhend & Choudhari, 2026) and Hinglish-TOP (Agarwal et al., 2023) with a multiple-negatives ranking loss and near-miss hard negatives.
Part of the BCA (Honours, AI/ML) dissertation Verified Semantic Caching for Hinglish and Code-Mixed LLM Queries, Symbiosis Institute of Computer Studies and Research (SICSR), Pune. Team: Mohammad Jamal Yusuf, Myshaa Khan, Dhiren (SICSR, Pune); mentor: Prof. Arun Kumar.
Results
| Test set: right cached question ranked first | BGE-M3 | this model |
|---|---|---|
| Customer support, Hinglish | 61.3% | 84.9% |
| Customer support, English | 87.7% | 89.6% |
| Customer support, Hindi (Devanagari) | 80.7% | 77.4% |
| Hinglish-TOP (unseen intents) | 52.7% | 65.0% |
Splits are by intent: no test intent was seen in training. Full pipeline, ablations and negative results
are in the project repository and the W&B project hinglish-semcache-dissertation/hinglish-semcache.
Use
from sentence_transformers import SentenceTransformer
m = SentenceTransformer('jamal10110/bge-m3-hinglish-semcache')
v = m.encode(['account ka password reset kaise kare?', 'How do I reset my password?'], normalize_embeddings=True)
print(v[0] @ v[1])
Limitations
The customer-support benchmark is synthetic and LLM-assisted, and small (about 265 test questions per language); its near-miss probes were drafted by the authors with LLM assistance and have no Devanagari versions. Hindi (Devanagari) retrieval is slightly lower than with the original model. Hinglish-TOP is human-written.