ModernBERT-base-mlm-wiki-code

Model Summary

ModernBERT-base-mlm-wiki-code is a continued pre-trained version of answerdotai/ModernBERT-base, further trained on a large mixed corpus of English natural language text and multi-language source code using Masked Language Modeling (MLM).

The model was trained with a challenging 30% masking probability (vs. the standard 15% in BERT) over a 2048 token context window, achieving a final perplexity of 1.9507 — indicating strong and confident language + code understanding across both domains.

---

Evaluation Results

Evaluation was performed separately on each domain to understand per-domain performance.

MLM Probability: 30% — Context Length: 2048 tokens

---

Model Details

---

Usage

Installation

pip install transformers torch

Load the Model

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
model = AutoModelForMaskedLM.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
model.config.reference_compile = False

Fill-Mask — Natural Language

from transformers import pipeline

pipe = pipeline("fill-mask", model="kd13/ModernBERT-base-mlm-wiki-code")

result = pipe("The capital of France is [MASK].")
for r in result:
    print(f"{r['token_str']:15s} → {r['score']:.4f}")

Fill-Mask — Source Code

from transformers import pipeline

pipe = pipeline("fill-mask", model="kd13/ModernBERT-base-mlm-wiki-code")

result = pipe("def fibonacci(n): return n if n <= 1 else fibonacci(n-1) [MASK] fibonacci(n-2)")
for r in result:
    print(f"{r['token_str']:15s} → {r['score']:.4f}")

Feature Extraction (Embeddings)

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
model = AutoModel.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")

text = "def quicksort(arr): return arr if len(arr) <= 1 else ..."
inputs = tokenizer(text, return_tensors="pt", max_length=2048, truncation=True)

with torch.no_grad():
    outputs = model(**inputs)

# CLS token embedding — shape: (1, 768)
embedding = outputs.last_hidden_state[:, 0, :]
print(embedding.shape)

---

Citation

If you use this model, please cite the original ModernBERT paper:

@article{modernbert2024,
  title   = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author  = {Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
  year    = {2024},
  url     = {https://arxiv.org/abs/2412.13663}
}

---

Acknowledgements

• Base model by Answer.AI
• Training data from WikiText and BigCode The Stack V2