English注册

johannhartmann/mlateon-esci-product-search

来源：🤗feature-extractionapache-2.0307M 参数1.2 GBsafetensors✓ 5 个校验和20 天前更新
或提交种子

mLateOn fine-tuned for product search (ESCI)

A ColBERT-style late-interaction retriever for matching free-text queries to products, fine-tuned from lightonai/mLateOn-unsupervised on the public Amazon Shopping Queries Dataset (ESCI, US locale). Full fine-tuning, not an adapter.

Built for the talk "Lokale Modelle selbst trainieren" (Mayflower, 2026). The motivating case: matching public tenders against a catalog of about half a million products, where dense and BM25 retrieval find only a small fraction of the relevant products. Customer data cannot be published, so this model demonstrates the same training recipe on ESCI.

Usage

# pip install "sentence-transformers[train]==6.1.0" "transformers>=5,<6"
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("johannhartmann/mlateon-esci-product-search")
q = model.encode_query(["waterproof hiking boots women"])
d = model.encode_document(["Product: KEEN Women's Targhee III Mid Height Waterproof Hiking Boot ...",
                           "Product: Stainless steel cocktail shaker set ..."])
print(model.similarity(q, d))   # MaxSim scores

Queries are prefixed with [Q] , documents with [D] automatically (configured prompts). Trained with 32 query tokens and 256 document tokens. Documents were rendered as Product: <title> plus brand, color, features (bullet points) and description fields (see the notebook under training/). For large catalogs, use an index (e.g. PLAID) and rerank with exact MaxSim.

Results

All 8,955 official ESCI test queries (US), each searched exhaustively over all 482,105 US products (exact MaxSim, no approximate index). nDCG uses the graded ESCI gains (E=1, S=0.1, C=0.01, I=0).

Method Recall@10 Recall@100 nDCG@10 MRR@10
BM25 (bm25s) 0.296 0.581 0.318 0.460
Dense single-vector (mDenseOn) 0.334 0.640 0.367 0.533
mLateOn-unsupervised (base) 0.362 0.669 0.392 0.549
lightonai/mLateOn (LightOn's supervised release) 0.388 0.697 0.423 0.592
this model 0.406 0.723 0.443 0.606

Versus the base model: Recall@100 +0.054 (95% bootstrap CI +0.050 to +0.058), nDCG@10 +0.051 (CI +0.048 to +0.055).

Results with LLM-completed labels (unjudged hits counted)

ESCI judges only ~16 products per query, so on a full-catalog search most top-10 hits were never judged and the table above counts them as misses (56–65% of the top 10 in the sample below). The numbers above are therefore lower bounds. On a fixed sample of 1,000 of the test queries, every unjudged top-10 hit of BM25, the base model and this model was labelled by an LLM judge (Claude Haiku 4.5, ESCI label definitions, text of title/brand/color/bullets/description); human ESCI labels were kept wherever they exist. Judgments, calibration set, prompt and raw API responses: johannhartmann/esci-llm-judged-top10.

These completed numbers rely on LLM labels, not human annotation. Judge calibration on 1,500 blind human-labelled pairs of the same queries: accuracy 0.61 (4 classes), Cohen's κ 0.38; Exact vs. rest accuracy 0.72, κ 0.43. On the 1,000 pairs from the top-10 pool the judge's share of Exact labels matches the humans' per method within 5 points (BM25 −4.2, base −1.8, this model +1.3).

Top-10 over all 482,105 products, 95% bootstrap CIs over queries; nDCG@10 with ESCI gains and the ideal ranking from all known labels of a query (completed values are only comparable with each other, not with the ESCI-only column).

Method nDCG@10 ESCI-only nDCG@10 LLM-completed P@10 (Exact) LLM-completed P@10 (Exact or Substitute) LLM-completed
BM25 (bm25s) 0.322 (0.303–0.341) 0.492 (0.473–0.510) 0.366 (0.348–0.384) 0.626 (0.606–0.646)
mLateOn-unsupervised (base) 0.392 (0.373–0.411) 0.581 (0.564–0.599) 0.440 (0.422–0.457) 0.691 (0.673–0.709)
this model 0.450 (0.430–0.470) 0.686 (0.670–0.702) 0.523 (0.504–0.541) 0.805 (0.788–0.821)

This model vs. base, paired bootstrap on the same 1,000 queries: nDCG@10 +0.058 ESCI-only (+0.047 to +0.070) vs. +0.105 LLM-completed (+0.092 to +0.118); P@10 (Exact) +0.035 vs. +0.083; P@10 (Exact or Substitute) +0.049 vs. +0.115. With completed labels nDCG@10 is better on 68% and worse on 27% of the queries.

Training

  • Data: ESCI US train split (18,961 queries; 1,000 further queries held out as dev for checkpoint selection). Positives: Exact products. Negatives: labelled Complement/Irrelevant plus hard negatives mined with the base model by exact MaxSim over the whole catalog (top 500, 1% relative margin, known positives/substitutes excluded). Substitute is never used as a negative. 67,697 triplets.
  • Loss: CachedMultiVectorMultipleNegativesRankingLoss, scale 30 (the library default of 1.0 degraded dev nDCG@10 from 0.548 to 0.470; 10 → 0.569; 30 → 0.580).
  • Batch 128 (mini-batch 16), lr 1e-5, 1 epoch = 529 steps, bf16; best dev checkpoint at step 335. 75 min on an RTX A6000, peak 8.0 GiB. Full recipe in training_recipe.json.

Limitations

  • English product search on Amazon-style titles; other languages and domains (e.g. technical tender texts) need their own fine-tuning. The notebook has a path for your own query/positive(/negative) data.
  • Evaluation covers ESCI only. Measured with human ESCI labels alone the improvement over the base is moderate (+5 points nDCG@10), but that is a lower bound because most top-10 hits are unjudged; with LLM-completed labels on a 1,000-query sample it is +10.5 points. The completed numbers depend on an LLM judge with moderate agreement with the human annotators (κ 0.38) – see the dataset card for calibration and limitations.

License and attribution

Model: Apache-2.0. Base model lightonai/mLateOn-unsupervised: Apache-2.0. Training and evaluation data: Shopping Queries Dataset (ESCI), Apache-2.0 (Reddy et al., 2022, Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search).