Technical Domain Classifier

This model is a fine-tuned version of allenai/specter2_base for multilabel scientific domain classification aligned with ERC panel taxonomy.
It achieves the following results on the held-out test set:

Best validation loss: 0.0372
Micro F1: 0.74

Model description

This model is a fine-tuned variant of SPECTER2 (allenai/specter2_base) adapted for multilabel classification of scientific documents into S3 technical domains.

The model takes as input the title and abstract of a scientific project or publication and predicts one or more domains.
Since scientific outputs may legitimately span multiple domains, the model is trained using sigmoid activation with binary cross-entropy loss, allowing independent assignment of multiple labels.

Key characteristics

Base model: allenai/specter2_base
Task: multilabel document classification
Labels: 28 S3 technical domains
Activation: sigmoid (independent scores per label)
Loss: BCEWithLogitsLoss
Output: list of predicted panels with associated probabilities
Decision threshold: 0.5 (tunable)

This model enables automatic tagging aligned with the S3 technical domains structure.

Intended uses & limitations

Intended uses

This model is designed for:

Automatic assignment of S3 technical domains

The model supports:

title only
abstract only
title + abstract (recommended)

Limitations

Labels are derived from automatically annotated data
Class imbalance may affect recall for underrepresented panels
The model does not encode explicit hierarchical relationships between panels

How to use

Training and evaluation data

Training data

Scientific documents with S3 technical domains annotations
Inputs:
- title
- abstract
Task type: multilabel classification

Dataset characteristics

Property	Value
Documents	~16k
Labels	28 panels
Input fields	Title, Abstract
Task type	Multilabel
License	Dataset-dependent

Training procedure

Preprocessing

Input text constructed as:

title + ". " + abstract
Tokenization using the SPECTER2 tokenizer
Maximum sequence length: 512 tokens

Model

Base model: allenai/specter2_base
Classification head: linear → sigmoid
Loss function: BCEWithLogitsLoss
Predictions: independent probability per label

Training hyperparameters

Hyperparameter	Value
Learning rate	5e-5
Train batch size	4
Eval batch size	8
Epochs	5
Weight decay	0.01
Optimizer	AdamW
Metric for best model	Micro F1

Training results

Epoch	Training Loss	Duration seconds
1	0.1618	234.58
2	0.0857	234.35
3	0.0575	234.92
4	0.0365	235.46
5	0.0217	236.22

Evaluation results (multilabel test set)

Panel	Precision	Recall	F1-score	Support
2D materials	0.90	0.65	0.75	71
3D printing materials	0.89	0.50	0.64	48
Additive manufacturing	0.69	0.75	0.72	79
Artificial Intelligence and machine learning	0.55	0.90	0.69	176
Big-data and analytics	0.66	0.67	0.66	218
Biomaterials, biosensors, bioactuators and bioactivators	0.76	0.78	0.77	117
Biomedical, pharmaceutical, pharmacological technologies	0.90	0.78	0.84	278
Biorobotics	0.92	0.75	0.82	59
Biotechnology for health	0.55	0.93	0.69	142
Business models and digital creativity	0.60	0.74	0.66	323
Cyber connectivity and 5G	0.77	0.79	0.78	119
Cyber-security, blockchain	0.88	0.92	0.90	120
Development of energy-saving solutions	0.63	0.76	0.69	143
Development of production and organisational processes based on intelligent sensors/devices	0.68	0.49	0.57	126
Environmental resilience and pollution	0.71	0.76	0.74	167
Exploitation of renewable energy sources	0.83	0.80	0.81	113
Genomics	0.90	0.86	0.88	42
Green propulsion technologies	0.95	0.52	0.67	69
High-performance and sustainable composite materials	0.77	0.64	0.70	181
Industrial biotechnology	0.52	0.88	0.65	58
Internet of the things and services	0.68	0.78	0.73	228
Nanomaterials and nanotechnology	0.82	0.89	0.85	266
Neuroscience	0.72	0.93	0.81	67
Nutraceutics	0.36	0.94	0.52	17
Photonic, micro and nanoelectronic solutions	0.79	0.88	0.84	259
Robotic, mechatronic and automation solutions	0.42	0.94	0.58	144
Technologies for decarbonisation	0.74	0.86	0.80	225
Technologies for the circular economy and bioeconomy	0.87	0.76	0.81	123

Overall performance

	Precision	Recall	F1-score	Support
Micro avg	0.70	0.79	0.74	2077
Macro avg	0.73	0.78	0.73	2077

Framework versions

Transformers: 4.57.x
PyTorch: 2.8.0
Datasets: 3.x
Tokenizers: 0.22.x

SIRIS-Lab/Toscana-TechDomain-classifier