DistilBERT Base Uncased - Domain Adaptation, Fine-Tuned on IMDB Movie Reviews (Standard Subword Masking)

This model is a domain-adapted version of distilbert-base-uncased fine-tuned on the IMDB movie reviews dataset for Masked Language Modeling (MLM).

By adapting the base model to informal, domain-specific film terminology, its language modeling performance improved significantly over the base checkpoint.

Model Description

• Base Model: distilbert-base-uncased
• Task: Domain Adaptation / Masked Language Modeling (Fill-Mask)
• Dataset: stanfordnlp/imdb (all splits concatenated and chunked into 128-token blocks)
• Objective: Adapt general language representations to informal, domain-specific movie review language and vocabulary co-occurrences.

Performance & Results

• Evaluation Metric: Cross-Entropy Loss & Perplexity
• Pre-training Perplexity: 23.14
• Fine-tuned Perplexity: 9.91

Note: For Masked Language Modeling, the loss evaluated during training is the cross-entropy over masked tokens. A standard metric for measuring domain adaptation is Perplexity, calculated as exp(eval_loss). Lower perplexity indicates that the model is less "surprised" by text from the target domain.

Training Details

• Training Dataset Size: 50,000 sequence chunks (128 max token length, full packing / zero padding)
• Evaluation Dataset Size: 10,000 sequence chunks (Official IMDB test split)
• Hardware: NVIDIA T4 GPU (Google Colab)
• Collator Strategy: Standard Subword Masking (DataCollatorForLanguageModeling, 15% probability for dynamic masking)

Note: A Whole-Word Masking (WWM) Collator variant is available in distilbert-base-uncased-finetuned-imdb-wwm, together with its results and a discussion of the comparison.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• weight_decay: 0.01
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3.0
• precision: FP16 mixed-precision training (fp16=True, native PyTorch AMP)

Training results

Framework versions

• Transformers 5.13.1
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

Usage

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="AlexStamp/distilbert-base-uncased-finetuned-imdb-standard")

# Example domain completion
result = fill_mask("This movie was completely [MASK] and worth every penny!")
for prediction in result:
    print(f"{prediction['token_str']}: {prediction['score']:.4f}")