Model Card for SzegedAI/bert-medium-mlsm

This medium-sized BERT model was created using the Masked Latent Semantic Modeling (MLSM) pre-training objective, which is a sample efficient alternative for classic Masked Language Modeling (MLM).
During MLSM, the objective is to recover the latent semantic profile of the masked tokens, as opposed to recovering their exact identity.
The contextualized latent semantic profile during pre-training is determined by performing sparse coding of the hidden representation of an already pre-trained model (a base-sized BERT model in this particular case).

Model Details

Model Description

• Developed by: SzegedAI
• Model type: transformer encoder
• Language: Engish
• License: MIT

Model Sources

• Repository: https://github.com/szegedai/MLSM
• Paper: Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling

How to Get Started with the Model

The pre-trained model can be used in the usual manner, e.g., for fine tuning on a particular sequence classification task, invoke the code:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('SzegedAI/bert-medium-mlsm')
model = AutoModelForSequenceClassification.from_pretrained('SzegedAI/bert-medium-mlsm')

Training Details

Training Data

The model was pre-trained using a 2022 English Wikipedia dump pre-processed with wiki-bert-pipeline.

Training Procedure

Preprocessing

Training Hyperparameters

Pre-training was conducted with a batch size of 32 sequences and a gradient accumulation over 32 batches, resulting in an effective batch size of 1024.
A total of 300,000 update steps were performed using the AdamW optimizer with a linear learning rate scheduling having a peak learning rate of 1e-04. A maximum sequence length of 128 tokens was employed over the first 90% of the pre-training, while for the final 10% of the pre-training, the maximum sequence length was increased to 512 tokens.

• Training regime: fp32

Evaluation

Metrics

The model was evaluated on GLUE tasks and CoNLL2003 for named entity recognition.

Results

The evaluation result after fine-tuning the given model on a wide range of tasks.
On each tasks 10 different fine-tuning were performed, during which the only difference was the random initialization of the task-specific classification head. Both the average and the standard deviation are displayed below on each tasks.

Summary

This model was more sample efficient and reached practically the same average performance as an alternatively pre-trained language model of 2.5 times more parameter (of base size) that was pre-trained using the classical MLM objective.

Environmental Impact

• Hardware Type: RTX A6000
• Hours used: 300
• Carbon Emitted: cca. 42 kg CO2 eq. (based on Machine Learning Impact calculator)

Citation

The pre-training objective is introduced in the ACL Findings paper Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling.

BibTeX:

@inproceedings{berend-2023-masked,
    title = "Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling",
    author = "Berend, G{\'a}bor",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.876",
    pages = "13949--13962",
    abstract = "In this paper, we propose an alternative to the classic masked language modeling (MLM) pre-training paradigm, where the objective is altered from the reconstruction of the exact identity of randomly selected masked subwords to the prediction of their latent semantic properties. We coin the proposed pre-training technique masked latent semantic modeling (MLSM for short). In order to make the contextualized determination of the latent semantic properties of the masked subwords possible, we rely on an unsupervised technique which uses sparse coding. Our experimental results reveal that the fine-tuned performance of those models that we pre-trained via MLSM is consistently and significantly better compared to the use of vanilla MLM pretraining and other strong baselines.",
}