ms-marco-MiniLM-L6-v2-finetuned-scidocs

This model is a fine-tuned version of cross-encoder/ms-marco-MiniLM-L6-v2 on GreenNode/scidocs-reranking-vn dataset. It achieves the following results on the evaluation set:

• Loss: 0.9168
• Accuracy: 0.9175
• F1: 0.7525

Model description

This model is a Cross-Encoder for Text Ranking based on the cross-encoder/ms-marco-MiniLM-L6-v2 architecture. It has been fine-tuned to assess the semantic relevance between a query and a document, outputting a score that indicates how likely the document contains the answer or relevant information for the query.

Unlike Bi-Encoders (which map text to vector space), this Cross-Encoder processes the query and document simultaneously, allowing for deep semantic interaction and superior ranking performance, particularly for scientific and technical domains.

• Task: Semantic Search / Re-Ranking
• Base Model: cross-encoder/ms-marco-MiniLM-L6-v2
• Language: English (Scientific/Academic domain focus)
• Input: A pair of texts (Query, Document)
• Output: A single logit/score (High score = Relevant)

Intended uses & limitations

Intended Uses

• Information Retrieval (RAG): Re-ranking the top-100 documents retrieved by a vector search (e.g., BM25 or FAISS) to improve the precision of the top-10 results.
• Scientific Literature Search: Specifically optimized for matching research queries with relevant paper titles/abstracts.
• Question Answering: Filtering irrelevant context passages before feeding them to a generative LLM.

Limitations

• Domain Specificity: The model was trained primarily on scientific and academic text (SciDocs). It may exhibit "keyword bias" in general domains (e.g., associating technical terms like "Optimistic" solely with computer science rather than psychology).
• Latency: As a Cross-Encoder, it requires a full forward pass for every query-document pair. It is computationally expensive and not suitable for searching millions of documents directly. It should be used as the second stage in a Retrieve-then-Rerank pipeline.
• Max Sequence Length: Limited to 512 tokens. Documents longer than this will be truncated, potentially losing relevant information at the end.

Training and evaluation data

The model was fine-tuned on the SciDocs / GreenNode Scidocs Reranking dataset.

• Data Structure: The dataset consists of scientific queries paired with "Positive" (cited/relevant) papers and "Negative" (irrelevant) papers.
• Class Imbalance: The training data was heavily imbalanced, containing approximately 19% Positive samples and 81% Negative samples.
• Preprocessing: The dataset was flattened from a list-based structure (1 query -> N documents) into individual training pairs (1 query -> 1 document).

Training procedure

The model was trained using the Hugging Face Trainer API with a custom weighted loss function to address class imbalance.

Hyperparameters

• Learning Rate: 2e-5
• Batch Size: 16
• Epochs: 3
• Optimizer: AdamW
• Precision: FP16 (Mixed Precision)
• Max Length: 512 tokens

Optimization Strategy

To prevent the model from collapsing into predicting "Not Relevant" (due to the 81% negative rate), we implemented a Weighted Binary Cross Entropy Loss.

• Loss Function: BCEWithLogitsLoss with pos_weight=5.0.
• This weighting (calculated based on dataset statistics) penalizes the model 5x more for missing a relevant document than for misclassifying an irrelevant one, ensuring high recall.

Performance

• Accuracy: ~92%
• F1 Score: ~0.75
• Validation Loss: ~0.916 The model achieved peak performance at Epoch 1 before overfitting.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.53.3
• Pytorch 2.6.0+cu124
• Datasets 4.4.1
• Tokenizers 0.21.2