Model Card for Indus (indus-sde-v0.1)

This model was further pre-trained on Science Discovery Engine (SDE) website data from nasa-smd-ibm-v0.1 with Masked Language Modelling task.

Model Details

• Base Model: nasa-impact/nasa-smd-ibm-v0.1
• Tokenizer: nasa-impact/nasa-smd-ibm-v0.1
• Parameters: 125M
• Pretraining Strategy: Masked Language Modeling (MLM)

Training Data

• Science Discovery Engine (SDE) Website Data

Training Procedure

• transformers Version: 4.47.1
• Strategy: Masked Language Modeling (MLM)
• Masked Language Model Probability: 15%
• Learning Rate: 1e-5
• Batch Size: 16
• Epoch: 7

Dataset

• Total Data Size: 121,240
• Validation Data Size: 10% of total size
• Test Data Size: 10% of total size

Evaluation

• Train Mask Accuracy: {'top1': 0.833, 'top2': 0.8895, 'top3': 0.9071}
• Test Mask Accuracy: {'top1': 0.8211, 'top2': 0.8768, 'top3': 0.894}