Sentence Transformers

We are forking sentence-transformers/all-MiniLM-L6-v2 as it is similar to the targeting dataset and use case. For more details, please check the pre-trained model weight repository.

• https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
• Commit Hash: 7dbbc90392e2f80f3d3c277d6e90027e55de9125

Fine-tuning

• Fine-tune the model using a contrastive objective.
• Compute the cosine similarity from each possible sentence pairs from the batch.
• Then apply the cross entropy loss by comparing with true pairs.

Hyper parameters

• Train the model during 100k steps using a batch size of 1024 (128 per TPU core).
• Use a learning rate warm up of 500.
• The sequence length was limited to 128 tokens.
• Used the AdamW optimizer with a 2e-5 learning rate.
• The full training script is accessible in this current repository: train_script.py.

Performance

Datasets