EVE-Rerank-1.0-Legal

Korean cross-encoder reranker for legal precedent retrieval, fine-tuned from dragonkue/bge-reranker-v2-m3-ko.

Stage 2 of a retrieval pipeline: it reorders the top 20 candidates from EVE-Embed-1.1-Legal.

Scored on EVE-Bench-Legal-2.0: 5,998 practitioner-style queries, retrieval over the full 59,786-precedent corpus, graded multi-gold relevance labels.

P@1 is the number that matters in use: the pipeline puts a genuinely relevant precedent first for 93.6% of queries, against 89.4% for retrieval alone.

All gaps are significant under a paired bootstrap over the same queries (10,000 resamples):

Usage

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("ysmeta/EVE-Rerank-1.0-Legal", max_length=512)

query = "계약금을 이미 받았는데도 상대방이 계약을 취소할 수 있나요?"
candidates = [...]                      # top-20 docs from EVE-Embed-1.1-Legal

scores = reranker.predict([[query, d] for d in candidates])
ranked = [d for _, d in sorted(zip(scores, candidates), key=lambda x: -x[0])]

Reranking 20 candidates costs 20 cross-encoder forward passes per query. Going beyond top-20 buys little: 93.6% of queries already have a relevant precedent at rank 1.

What made this work: relevant candidates are positives, not noise

A first attempt (v1) trained against verified hard negatives — candidates that an LLM judge confirmed do not answer the query — and discarded the ~30% it judged relevant. That is the recipe that works for the embedding model, where unverified negatives cost −36.5%. For a reranker it was optimising the wrong target.

Legal precedents cluster by 쟁점, so a query typically has 7.20 genuinely relevant precedents, not one. The task is to rank all of them above the irrelevant ones. Discarding the relevant siblings meant v1 never trained on the decision it is scored on. Its evaluation curve fell from its very first measurement while the loss went down.

v2 turns every judged-relevant candidate into its own training row sharing the query's irrelevant candidates as negatives:

Same base model, same data, same judge — a 6× larger margin from changing what counts as a positive. The curve inverted too: v1 ran 0.8822 → 0.8335 (best at its first eval), v2 ran 0.8152 → 0.8362 (best at its last).

Training

• Base: dragonkue/bge-reranker-v2-m3-ko (Apache-2.0)
• Data: 23,600 queries → 41,706 rows (1.77 positives per query, 4 negatives each), from EVE-Train-Legal-Rerank-1.0
• Candidates come from EVE-Embed-1.1's own top-15, judged by tencent/Hy3. A reranker trained on some other retriever's candidates solves a different problem than the one it is deployed into.
• Loss: CachedMultipleNegativesRankingLoss (listwise softmax over 1 positive + 4 negatives)
• LR: 2e-6, 1 epoch, effective batch 32, max_length 512, bf16, 1,304 steps (~76 min)
• Checkpoint selection: load_best_model_at_end on a held-out graded slice, with eval_on_start=True so the untrained reference sits inside the curve
• Hardware: 1 × NVIDIA H200

v1 used lr 1e-5 and degraded monotonically from step 100 — too high for an already-trained reranker.

Limitations

• Tuned for this pipeline. Candidates come from EVE-Embed-1.1; behaviour on another retriever's candidate distribution is untested.
• Relevance labels are LLM-generated, not lawyer-reviewed, and inherit the judge's reading of what "answers" a legal question.
• 판례 only, no statute text (법령 조문). Case types: 민사 46.3% · 형사 21.9% · 일반행정 14.1% · 세무 11.9% · 특허 4.3% · 가사 1.6%.
• Benchmark queries are LLM-generated from their source precedent and carry traces of it, so part of the measured score reflects trace-matching rather than legal understanding. See the benchmark card for the evidence.
• If you would rather not add a second model, dragonkue/bge-reranker-v2-m3-ko gets you from 0.7253 to 0.7506 for free — 54% of this model's gain.
• Not legal advice. Ranking surfaces precedents; it does not interpret them.

License

Apache-2.0, inherited from the base model.