muril-hinglish-cache-verifier
MuRIL fine-tuned as a cross-encoder verifier: given (new question, cached question) it outputs the probability that the cached answer can be reused. Used to block wrong cache hits in a verified semantic cache for Hinglish.
Part of the BCA (Honours, AI/ML) dissertation Verified Semantic Caching for Hinglish and Code-Mixed LLM Queries, Symbiosis Institute of Computer Studies and Research (SICSR), Pune. Team: Mohammad Jamal Yusuf, Myshaa Khan, Dhiren (SICSR, Pune); mentor: Prof. Arun Kumar.
Results
| Test ROC-AUC (correct vs wrong best match) | English | Hinglish | Hindi |
|---|---|---|---|
| this model | 0.81 | 0.81 | 0.72 |
| bge-reranker-v2-m3 (not fine-tuned) | 0.82 | 0.75 | 0.72 |
About 5 ms per check on an Apple M2 CPU.
Splits are by intent: no test intent was seen in training. Full pipeline, ablations and negative results
are in the project repository and the W&B project hinglish-semcache-dissertation/hinglish-semcache.
Use
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained('jamal10110/muril-hinglish-cache-verifier')
m = AutoModelForSequenceClassification.from_pretrained('jamal10110/muril-hinglish-cache-verifier')
x = tok(['wallet ka PIN kaise reset karu?'], ['How do I reset my account password?'], return_tensors='pt')
print(torch.sigmoid(m(**x).logits[0, -1]).item()) # low = do not reuse
Limitations
The customer-support benchmark is synthetic and LLM-assisted, and small (about 265 test questions per language); its near-miss probes were drafted by the authors with LLM assistance and have no Devanagari versions. Hindi (Devanagari) retrieval is slightly lower than with the original model. Hinglish-TOP is human-written.