This model is a retriever for ROR set point matching, distilled from OpenAlex's Jev judgments. It finds candidate ROR records for an affiliation string.
It learns from the teacher judgments of openalex-affiliation-matcher v3.0.0. These judgments come from Jev, a closed commercial decision model from TypeSafe AI. The matcher uses ror-set-point-retriever-e5-base, which learns from the judgments of StartLux-Decision v1.1 27B Instant, a model with open weights. Its card compares the two retrievers.
This repository is a copy of revision 3f011b6 of ror-set-point-retriever-e5-base. The set model ror-set-point-set-model-mmbert-base-openalex-jev-distill reads the candidates of this model.
Part of the ROR set point matching collection.
The model changes text into vectors. It changes an affiliation string into a query vector, and the text of each ROR record into a record vector. A high dot product means a good candidate. The matcher searches all 143,284 records and keeps the top 32. It adds the top 10 records of a lexical index, and a set model then selects the records that the string names.
The model is one shared encoder with mean pooling. The output vectors have a length of 1. The weights are fp16.
from sentence_transformers import SentenceTransformer
from datasets import load_dataset
model = SentenceTransformer("adambuttrick/ror-set-point-retriever-e5-base-openalex-jev-distill")
records = load_dataset("adambuttrick/ror-set-point-matching-train-validation-test", "ror_records", split="train")
model.max_seq_length = 128 # record texts: 128 tokens, as in training
r = model.encode(records["text"], prompt_name="passage", batch_size=256, convert_to_tensor=True)
model.max_seq_length = 512 # strings: up to 512 tokens
q = model.encode(["Dept. of Physics, University of Tokyo, Japan"], prompt_name="query", convert_to_tensor=True)
top = (q @ r.T).topk(32)
candidates = [records[int(i)]["ror_id"] for i in top.indices[0]]
The prompts are "query: " for strings and "passage: " for records.
The teacher judgments come from the dataset ror-set-point-matching-train-validation-test-openalex-jev-distill. All other training data comes from the dataset ror-set-point-matching-train-validation-test. The exact rows are in training_data/.
| Part | Config and split | Hard negatives | Rows |
|---|---|---|---|
| Teacher | teacher_judgments train (Jev): 167,533 strings, one row for each record with p ≥ 0.5 |
The records of the same string with p < 0.1 | 304,626 |
| Labeled | affiliations_unseen_institutions train: 16,452 strings, 16,630 links. Each link is used 3 times |
The candidates of the baseline that are not labels | 49,890 |
| Synthetic | synthetic_non_latin train: 130,886 strings. The 2,871 strings that name a held-out record are not used |
Mined (see below) | 136,188 |
The training data has 490,704 rows. In each step, each string has its positive record and 3 hard negatives, drawn at random from the hard negatives of the row. All other records of the batch are also negatives, except the other labels of the string.
These teacher strings are not used:
The rows contain 8 strings (4 labeled and 4 synthetic) that are equal to affiliations validation strings. No test string is among them.
The first retriever, ror-set-point-retriever-e5-base-no-synthetic, searched each synthetic string and kept the top 64 records. The procedure read these records in rank order and dropped:
The first 10 records that remain are the negatives. The mean is 9.6 for each string. The labels in this step were the labels before the label rules of the dataset (see the synthetic_non_latin section of the dataset card).
| Setting | Value |
|---|---|
| Base model | intfloat/multilingual-e5-base, revision d12875059715 |
| Loss | InfoNCE, temperature 0.02, over all records of the batch |
| Batch | 128 strings, each with 1 positive and 3 hard negatives |
| Epochs | 1 (3,833 steps) |
| Optimizer | AdamW, learning rate 2e-5, weight decay 0.01 |
| Schedule | Linear warm-up for 5% of the steps, then linear decay |
| Precision | bf16 autocast |
| Gradient clipping | 1.0 |
| Maximum length | 96 tokens for strings and 128 tokens for records. The search uses 512 tokens for strings |
| Gradient checkpointing | Off |
| Seed | 1363 |
| Hardware | 1 × NVIDIA H200, 17 minutes (996 s) |
Recall of the gold links in the top 32 records, in percent, on affiliations_unseen_institutions. The validation and test splits contain strings for institutions that are not in the train split. "New" links go to records that no training string names. "Known" links go to the other records.
| Split | All links | Known | New | Links in non-Latin strings | Links to held-out institutions |
|---|---|---|---|---|---|
| test (1,957 strings, 2,620 links) | 96.3 | 93.7 | 97.2 | 97.9 | 98.2 |
| validation (1,914 strings, 3,292 links) | 92.8 | 95.3 | 91.9 | 99.2 | 91.1 |
"New" refers to the train split only: the teacher and synthetic strings can name a new record. The held-out institutions (1,255) have no string in the train split.
The test row comes from the search in search/topk_test_splits.jsonl.gz, and the validation row from the search of the training run (search/topk.jsonl.gz). For comparison, on the test split:
| Candidates | Records for each string | All links | Known | New | Links in non-Latin strings |
|---|---|---|---|---|---|
| This model, top 32 | 32.0 | 96.3 | 93.7 | 97.2 | 97.9 |
| The same, plus the lexical top 10 | 37.5 | 97.6 | 96.5 | 98.0 | 97.9 |
| The candidates of the baseline (lexical, dense, acronym and alias generators) | 18.0 | 94.6 | 95.6 | 94.3 | 85.2 |
Recall of the gold links on the other test splits, in percent:
| Split | Strings | This model, top 32 | Top 32 and the lexical top 10 |
|---|---|---|---|
affiliations test |
1,947 | 98.7 | 98.8 |
datacite test |
3,000 | 99.0 | 99.4 |
openalex_sample test |
40,000 | 98.6 | 98.9 |
In these tables, the lexical top 10 comes from an index that reads the institutions file of openalex-affiliation-matcher v3.0.0 and adds a bonus for the works count. The current matcher uses an index that reads the ROR records, with no such bonus.
The validation split contains all 272 links to CNRS (https://ror.org/02feahw73). Most of these strings name CNRS only with its acronym, inside the name of a laboratory. This decreases the recall on that split.
| Path | Content |
|---|---|
| Model files | The sentence-transformers model, with the prompts query and passage |
training_data/ |
The training folder: records.jsonl.gz (ror_id, text), train.jsonl.gz (the rows), queries.jsonl.gz (set, id, text: the strings to search) and manifest.json (counts and sha256) |
search/topk.jsonl.gz |
set, id, ids, scores: the top 64 records for every query in training_data/queries.jsonl.gz, from a search on one NVIDIA H200. The rows of ror-set-point-set-model-mmbert-base-openalex-jev-distill come from this file |
search/topk_test_splits.jsonl.gz |
The top 64 records for each string of the test splits affiliations_test, unseen_test, datacite_test and openalex_sample_test, from a search on one NVIDIA H200 |
logs/ |
The training log and summary |
conversion_check.json |
A check that this model gives the same vectors as the training code |
Each training row has the fields q (the string), pos (the positive record), negs (the hard negatives), pos_all (all labels of the string) and src (teacher, labelled or synthetic). The query sets are teacher, labelled, synthetic (whose id is the line in synthetic_non_latin) and the evaluation sets.
The evaluation sets have these names in the files:
| Name | Split |
|---|---|
unseen_validation |
affiliations_unseen_institutions validation |
unseen_test |
affiliations_unseen_institutions test |
affiliations_validation |
affiliations validation |
affiliations_test |
affiliations test |
datacite_test |
datacite test |
openalex_sample_test |
openalex_sample test |
The training log uses the names of the training run.
The weights come from intfloat/multilingual-e5-base and use the MIT license.
intfloat/multilingual-e5-base