Fine-tuned model from the PsyNamic project (code, webapp): Clinical Trial Phase.
Retrained for the revision (Nov 2026) together with the other PsyNamic models. Not shown in the PsyNamic webapp and not reported in the paper (excluded parameter, see Task). It supersedes STRIDE-lab/clinicalbert-clinical-trial-phase-20250221 (tag submission-v1, initial submission). All current models: PsyNamic collection.
allenai/scibert_scivocab_uncased
Clinical Trial Phase (multi-label: several labels per abstract possible)
Identifies the trial phase (I-IV) or whether it is non-applicable.
Not shown in the PsyNamic webapp and not reported in the paper. Clinical Trial Phase and Study Conclusion were excluded from the final PsyNamic parameters due to their limited informativeness (predominantly "unknown" or "not applicable" labels) and comparatively low inter-annotator agreement (S1 Appendix of the PsyNamic paper, section 4.2).
| ID | Label | Definition |
|---|---|---|
| 0 | Phase 1 | Tests safety, dosage range, and pharmacokinetics in 20–100 people (usually healthy) |
| 1 | Phase 2 | Tests effectiveness and optimal dosing; 100–300 patients; often randomized and single- or double-blind. |
| 2 | Phase 3 | Large-scale testing (hundreds–thousands) to confirm safety and effectiveness before approval, randomized and double-blind |
| 3 | Phase 4 | Post-marketing surveillance; monitors long-term safety, rare side effects, and real-world effectiveness. |
| 4 | Unknown | Fulfilling above definition but phase not explicitly (or implicitly) mentioned |
| 5 | Not Applicable | Not fulfilling the above definition, e.g., an fMRI study assessing the changes upon lsd or a survey among MDMA users or a systematic review. |
Multi-label: a sigmoid is applied per label, and every label with probability ≥ 0.16 is predicted. This is the threshold used in the PsyNamic webapp. It was tuned on the validation set with a pre-specified rule: start at the threshold with maximal weighted F1, then lower it as long as weighted precision stays within 0.05 of its value there.
Five base models (BiomedBERT-abstract, BioBERT, BioLinkBERT, ClinicalBERT, SciBERT) were fine-tuned with identical hyperparameters. The checkpoint is the one with the lowest validation loss (early stopping), and the base model was selected on the validation set (weighted F1). The test set was used once, after selection.
Held-out test set (96 abstracts), tuned threshold as in the webapp, 95% bootstrap CIs (1,000 BCa resamples).
| Metric (test) | Value [95% CI] |
|---|---|
| Weighted F1 | 0.777 [0.701–0.832] |
| Weighted precision | 0.691 [0.597–0.755] |
| Weighted recall | 0.887 [0.816–0.938] |
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "STRIDE-lab/scibert-clinical-trial-phase-20261106"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
inputs = tokenizer("<title>. <abstract>", truncation=True, max_length=512, return_tensors="pt")
logits = model(**inputs).logits
probs = torch.sigmoid(logits)[0]
pred = [model.config.id2label[i] for i, p in enumerate(probs) if p >= 0.16]
{
"data": "<private_dataset>",
"task": "Clinical Trial Phase",
"model": "scibert",
"cross_val": false,
"batch_size": 8,
"learning_rate": 5e-05,
"weight_decay": 0.01,
"lr_scheduler": "linear",
"warmup_ratio": 0.1,
"epochs": 30,
"dropout": 0.1,
"early_stopping_patience": 5,
"gradient_clipping": 1.0,
"max_length": 512,
"threshold": 0.5,
"is_multilabel": true
}
threshold (0.5) is only the cut-off used to compute the validation metrics during training. It does not affect the model weights or the checkpoint choice (lowest validation loss). Predictions in the webapp and the test performance above use the tuned threshold 0.16 (see Prediction rule).
allenai/scibert_scivocab_uncased