ror-set-point-retriever-e5-base

This model is the retriever for ROR set point matching. It finds candidate ROR records for an affiliation string. The matcher uses this model.

The model learns from the judgments of StartLux-Decision v1.1 27B Instant, a decision model with open weights by StartLux Labs. ror-set-point-retriever-e5-base-openalex-jev-distill is a retriever that learns from Jev's judgments of the same pairs (see Evaluation).

Part of the ROR set point matching collection. The code is in ror-set-point-matching on GitHub.

What the model does

The model changes text into vectors. It changes an affiliation string into a query vector, and the text of each ROR record into a record vector. A high dot product means a good candidate. The matcher searches all 143,284 records and keeps the top 32. It adds the top 10 records of a lexical index. The set model ror-set-point-set-model-mmbert-base-mask then selects the records that the string names.

The model is one shared encoder with mean pooling. The output vectors have a length of 1. The weights are fp16.

How to use

from sentence_transformers import SentenceTransformer
from datasets import load_dataset

model = SentenceTransformer("adambuttrick/ror-set-point-retriever-e5-base")
records = load_dataset("adambuttrick/ror-set-point-matching-train-validation-test", "ror_records", split="train")

model.max_seq_length = 128          # record texts: 128 tokens, as in training
r = model.encode(records["text"], prompt_name="passage", batch_size=256, convert_to_tensor=True)
model.max_seq_length = 512          # strings: up to 512 tokens
q = model.encode(["Dept. of Physics, University of Tokyo, Japan"], prompt_name="query", convert_to_tensor=True)
top = (q @ r.T).topk(32)
candidates = [records[int(i)]["ror_id"] for i in top.indices[0]]

The prompts are "query: " for strings and "passage: " for records.

Training data

All training data comes from the dataset ror-set-point-matching-train-validation-test. The exact rows are in training_data/.

The training data has 344,245 rows. In each step, each string has its positive record and 3 hard negatives, drawn at random from the hard negatives of the row. When the row has fewer than 3, random records fill the list. All other records of the batch are also negatives, except the other labels of the string.

These strings are not used:

• teacher strings to which the teacher gives p ≥ 0.5 for a held-out record (19,455);
• teacher strings that are equal to an evaluation string (12);
• labeled and synthetic strings that are equal to an evaluation string (4 of each).

Thus no training row has a held-out record as a positive.

The teacher judged the (string, organization) pairs that Jev judged for openalex-affiliation-matcher v3.0.0. Platt scaling on 615,010 labeled pairs calibrates its p. The dataset card gives the question and the procedure.

Hard negatives for synthetic strings

A first retriever trained by the same procedure, but without the synthetic strings (208,061 rows, 1,625 steps). It searched each synthetic string and kept the top 64 records. The procedure read these records in rank order and dropped:

• the labels of the string;
• each record that is related to a label (parent, child, related, predecessor or successor);
• each record with the same name as the source record;
• each record with a score of 0.95 × the score of the source record, or more. The string probably names these records, but they have no label.

The first 10 records that remain are the negatives.

Training procedure

The training script saves the model, loads it again, and compares the vectors. The smallest cosine is 0.999997 for strings and 0.999998 for records (logs/training_summary.json).

Evaluation

Recall of the gold links in percent, on the test split of affiliations_unseen_institutions (1,957 strings, 2,620 links). Its strings name institutions that are not in the train split. "New" links go to records that no training string names. "Known" links go to the other records. "Held out" links go to the 1,255 held-out institutions.

"New" refers to the train split only: the teacher and synthetic strings can name a new record. The held-out institutions have no string in the train split.

Recall of the gold links on the other test splits, in percent:

The test figures come from the search in search/topk_test_splits.jsonl.gz.

The validation split of affiliations_unseen_institutions (1,914 strings, 3,292 links) contains all 272 links to CNRS (https://ror.org/02feahw73). Most of these strings name CNRS only with its acronym, inside the name of a laboratory. On this split, the recall of the top 32 is 95.5, and the matcher's candidates hold 96.8% of the links and 239 of the 272 links to CNRS. These figures come from the search of the training run (search/topk.jsonl.gz).

With the StartLux and Jev teachers

ror-set-point-retriever-e5-base-openalex-jev-distill learns from Jev's judgments of the same pairs. Recall in percent:

The rows of the Jev retriever also differ in these ways:

• The hard negatives of its labeled strings are the candidates of the baseline.
• Its rows contain 8 strings (4 labeled and 4 synthetic) that are equal to affiliations validation strings.
• The negatives of its synthetic strings come from the labels before the label rules of the dataset.

Files

Each training row has the fields q (the string), pos (the positive record), negs (the hard negatives), pos_all (all labels of the string) and src (teacher, labelled or synthetic). The query sets are teacher, labelled, synthetic (whose id is the line in synthetic_non_latin) and the evaluation sets.

The evaluation sets have these names in the files:

License and credits

The weights come from intfloat/multilingual-e5-base and use the MIT license.

The teacher judgments come from StartLux-Decision v1.1 27B Instant by StartLux Labs. The weights of the teacher (CC BY-NC 4.0) are not part of this repository.

Related repositories

• Dataset: ror-set-point-matching-train-validation-test.
• Collection: ROR set point matching.
• Set model: ror-set-point-set-model-mmbert-base-mask. Baseline: ror-set-point-pair-scorer-baseline.
• The pipeline with Jev's judgments: ror-set-point-retriever-e5-base-openalex-jev-distill and ror-set-point-set-model-mmbert-base-openalex-jev-distill.