MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.

This checkpoint is the original MobileBert Optimized Uncased English: uncased_L-24_H-128_B-512_A-4_F-4_OPT checkpoint.

This model was fine-tuned on html tags and labels using Fathom.

How to use MobileBERT in transformers

from transformers import pipeline

classifier = pipeline(
	"text-classification",
	model="vazish/mobile_bert_autofill"
)

print(
	classifier('<input class="cc-number" placeholder="Enter credit card number..." />')
)

Model Training Info

HyperParameters: {
'learning_rate': 0.000082,
'num_train_epochs': 12,
'weight_decay': 0.1,
'per_device_train_batch_size': 32,
}

More information on how the model was trained can be found here: https://github.com/mozilla/smart_autofill

Model Performance

Test Performance:
Precision: 0.97043
Recall: 0.96966
F1: 0.96921

                     precision    recall  f1-score   support

      CC Expiration      1.000     0.875     0.933        16
CC Expiration Month      1.000     0.972     0.986        36
 CC Expiration Year      0.973     0.973     0.973        37
            CC Name      1.000     0.968     0.984        31
          CC Number      0.942     0.980     0.961        50
    CC Payment Type      0.934     0.760     0.838        75
   CC Security Code      0.929     0.951     0.940        41
            CC Type      0.857     0.857     0.857        14
   Confirm Password      1.000     0.860     0.925        57
              Email      0.972     0.945     0.958        73
         First Name      0.833     1.000     0.909         5
               Form      0.950     0.974     0.962        39
          Last Name      0.833     1.000     0.909         5
       New Password      0.915     1.000     0.956        97
              Other      0.981     0.989     0.985      1235
              Phone      0.600     1.000     0.750         3
           Zip Code      0.939     0.969     0.954        32

           accuracy                          0.970      1846
          macro avg      0.921     0.945     0.928      1846
       weighted avg      0.970     0.970     0.969      1846