gte-modernbert-base fine-tuned for product search on Amazon ESCI
A bi-encoder for retrieving products from a shopping query. It is Alibaba-NLP/gte-modernbert-base (149M parameters, 768-d) fine-tuned on Amazon's Shopping Queries dataset (ESCI) with judged and mined hard negatives.
On 8,956 ESCI test queries, searching the full 482,105-product catalogue:
| Model | Recall@100 | Recall@1000 | NDCG@10 |
|---|---|---|---|
| BM25 (tuned) | 0.5775 | 0.7727 | 0.7062 |
| gte-modernbert-base, off-the-shelf | 0.5979 | 0.8097 | 0.7392 |
This model (run D, main) |
0.6483 | 0.8435 | 0.7543 |
| BM25 + this model, reciprocal rank fusion | 0.6867 | 0.8650 | 0.7409 |
Recall counts Exact products found in the top k of the whole catalogue. NDCG@10 orders each query's judged products, with gains Exact 1, Substitute 0.1, Complement 0.01, Irrelevant 0.
Project, code and full results: github.com/neha-nambiar/esci-product-search.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("nehajiya8/esci-gte-modernbert-bi")
model.max_seq_length = 256
query = "canon 5d mark iv"
products = [
"Canon EOS 5D Mark IV DSLR Body | Canon | Black | 30.4 MP full-frame CMOS sensor",
"Canon Battery Grip BG-E20 for the Canon 5D Mark IV Digital SLR Camera | Canon",
]
q = model.encode(query, normalize_embeddings=True)
p = model.encode(products, normalize_embeddings=True)
print(model.similarity(q, p))
Products were encoded as one text: title | brand | color | bullet points | description, empty fields skipped, cut to 256 tokens. Queries were cut to 64 tokens. Use cosine similarity.
Runs on other branches
Each run trains the same model on the same 17,678 pairs with the same settings. Only the negatives change.
| Branch | Negatives per pair | Recall@100 | NDCG@10 |
|---|---|---|---|
A |
in-batch only | 0.6204 | 0.7364 |
B |
+ 3 judged non-Exact products | 0.6396 | 0.7499 |
C |
+ 3 judged + 4 mined, filtered | 0.6428 | 0.7480 |
D (= main) |
+ 3 judged + 4 mined, unfiltered | 0.6483 | 0.7543 |
prefixed |
as D, with query: / document: prefixes |
0.6587 | 0.7581 |
Load a branch with SentenceTransformer("nehajiya8/esci-gte-modernbert-bi", revision="prefixed"). The prefixed model expects "query: " before queries and "document: " before product texts.
Training
- Data: ESCI English, Task 1 reduced. Positives are Exact query-product pairs from the train split; 17,678 pairs from 3,585 queries were kept so that every run could fill its negatives.
- Judged negatives: products ESCI labelled Substitute, Complement or Irrelevant for the query.
- Mined negatives: nearest neighbours of the query under the off-the-shelf model, with the query's Exact products excluded.
- Loss: multiple negatives ranking loss with GradCache (cached MNRL), batch 512 in chunks of 32, so every query sees all products in the batch.
- 1 epoch (35 steps), learning rate 2e-5, bf16, one RTX 4090 (about 12 minutes for this run).
Limits
One seed per run. The differences between C and D, and the NDCG gain from prefixes, are within noise. Evaluated on ESCI's US English products only.
Credits
Base model by Alibaba-NLP. Data: Amazon Shopping Queries dataset (ESCI), Apache-2.0. Trained with sentence-transformers.