Model Card for Indus (modernbert-sde-v0.2)
This model was further pre-trained on full Science Discovery Engine (SDE) website data from answerdotai/ModernBERT-base.
Model Details
- Base Model: answerdotai/ModernBERT-base
- Tokenizer: answerdotai/ModernBERT-base
- Parameters: 150M
- Pretraining Strategy: Masked Language Modeling (MLM)
Training Data
- Full Science Discovery Engine (SDE) Website Data
Training Procedure
- transformers Version: 4.48.3
- Strategy: Masked Language Modeling (MLM)
- Masking Strategy:
- Weighted Dynamic Masking based on Keyword Importance (YAKE) and Random Masking
- The idea for masking important keywords is to force the model to generalize for "science" keywords that gives a high signal for the document
- Masked Language Model Probability: 30%
- Weighted Dynamic Masking based on Keyword Importance (YAKE) and Random Masking
- Batch Size: 7
- Learning rate: 5e-5
- Warmup ratio: 0.1
Dataset
- Total Data Size: 545,717
- Validation Data Size: 10% of total size
- Test Data Size: 10% of total size