SIREN Screening Cross-encoder

A 3-class cross-encoder for systematic review screening that classifies query-document pairs as Relevant, Partial, or Irrelevant. Designed to rerank candidates from the siren-screening-biencoder.

Model Details

Label Definitions

Intended Use

Primary use case: Second-stage reranking in systematic review screening pipelines.

After retrieving candidates with a bi-encoder, use this cross-encoder to:

1. Rerank documents for better precision at top ranks
2. Classify relevance for triage (prioritize Relevant, defer Partial, skip Irrelevant)

Recommended pipeline:

1. Retrieve top-100 candidates with siren-screening-biencoder
2. Rerank with this cross-encoder
3. Use relevance labels to prioritize human screening

Usage

Sentence-Transformers CrossEncoder

from sentence_transformers import CrossEncoder

model = CrossEncoder("Praise2112/siren-screening-crossencoder")

# Pairs of (query, document)
pairs = [
    ("RCTs of aspirin in diabetic adults", "A randomized trial of aspirin in 5,000 diabetic patients showed..."),
    ("RCTs of aspirin in diabetic adults", "This cohort study examined statin use in elderly populations..."),
]

# Get 3-class scores
scores = model.predict(pairs)
print(scores)
# Output: array([[ 0.02,  0.15,  0.83],   # Relevant
#                [ 0.91,  0.07,  0.02]])  # Irrelevant

Transformers (Direct)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("Praise2112/siren-screening-crossencoder")
model = AutoModelForSequenceClassification.from_pretrained("Praise2112/siren-screening-crossencoder")

query = "RCTs of aspirin in diabetic adults"
document = "A randomized trial of aspirin in 5,000 diabetic patients showed reduced MI risk..."

inputs = tokenizer(
    query, document,
    padding=True,
    truncation=True,
    max_length=768,
    return_tensors="pt"
)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=-1)

print(f"Irrelevant: {probs[0, 0]:.3f}")
print(f"Partial: {probs[0, 1]:.3f}")
print(f"Relevant: {probs[0, 2]:.3f}")

# Get predicted label
label_id = probs.argmax().item()
labels = {0: "Irrelevant", 1: "Partial", 2: "Relevant"}
print(f"Prediction: {labels[label_id]}")

Scoring for Reranking

For reranking, convert 3-class probabilities to a single score:

def rerank_score(probs):
    """Convert 3-class probs to ranking score.

    Higher score = more relevant.
    Partial gets partial credit (1x), Relevant gets full credit (2x).
    """
    return probs[1] + 2 * probs[2]  # P(Partial) + 2 * P(Relevant)

# Example
probs = [0.02, 0.15, 0.83]  # [Irrelevant, Partial, Relevant]
score = rerank_score(probs)  # 0.15 + 2 * 0.83 = 1.81

Performance

Classification Accuracy

Reranking Impact (MRR@10)

General-purpose rerankers like BGE actually hurt performance on screening queries because they're optimized for topical relevance, not criteria matching.

Cross-encoder Transfer

This cross-encoder also improves other retrievers:

Training

This model was created by:

1. Fine-tuning on the siren-screening dataset with 3-class labels
2. SLERP merging encoder layers with the base model (t=0.2) to preserve generalization

Training details:

• Loss: Cross-entropy
• Batch size: 32 (16 x 2 gradient accumulation)
• Learning rate: 2e-5
• Epochs: 1
• Max length: 768 tokens

Limitations

• Synthetic queries, real documents: The queries and relevance labels are LLM-generated, but the documents are real PubMed articles
• English only: Trained on English PubMed content

Citation

@misc{oketola2026siren,
  title={SIREN: Improving Systematic Review Screening with Synthetic Training Data for Neural Retrievers},
  author={Praise Oketola},
  year={2026},
  howpublished={\url{https://huggingface.co/Praise2112/siren-screening-crossencoder}},
  note={Cross-encoder model}
}

License

Apache 2.0