Magic-BERT 50M MLM

A BERT-style transformer model trained for binary file understanding using masked language modeling (MLM). This model learns byte-level patterns in binary files, including magic bytes, headers, and structural patterns across 106 file types.

Why Not Just Use libmagic?

For intact files starting at byte 0, libmagic works well. But libmagic matches signatures at fixed offsets. Magic-BERT learns structural patterns throughout the file, enabling use cases where you don't have clean file boundaries:

• Network streams: Classifying packet payloads mid-connection, before headers arrive
• Disk forensics: Identifying file types during carving, when scanning raw disk images without filesystem metadata
• Fragment analysis: Working with partial files, slack space, or corrupted data
• Adversarial contexts: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading

Model Description

Magic-BERT uses a custom BERT architecture with absolute position embeddings, trained on binary file data using a byte-level BPE tokenizer. The MLM objective teaches the model to predict masked bytes given surrounding context, which implicitly learns file format structure.

Tokenizer

The tokenizer uses the Binary BPE methodology introduced in Bommarito (2025). The original Binary BPE tokenizers (available at mjbommar/binary-tokenizer-001-64k) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types including documents, images, audio/video, archives, and source code.

Intended Uses

Primary use cases:

• Fill-mask: Predicting missing bytes in binary files
• Magic byte and file signature recognition
• Feature extraction for downstream classification
• Research on binary file structure

Example tasks:

• Completing partial file headers
• Identifying file type from structure
• Anomaly detection in binary data

Detailed Use Cases

Network Traffic Analysis

When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Magic-BERT's structural understanding can identify file types from interior content.

Disk Forensics & File Carving

During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. Magic-BERT can score byte ranges for file type probability, helping identify carved fragments or validate carving results.

Incident Response

Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.

Embedded Content Detection

Files within files (email attachments, archive contents, OLE streams) may appear at arbitrary offsets. Embeddings from Magic-BERT enable similarity search: "find all chunks that look structurally like JPEG data" regardless of where they appear.

Training

Data

Trained on a diverse corpus of binary files spanning 106 MIME types, including:

• Documents (PDF, Office formats, OpenDocument)
• Images (PNG, JPEG, GIF, WebP, TIFF)
• Audio/Video (MP3, MP4, WebM, FLAC)
• Archives (ZIP, GZIP, 7z, TAR)
• Executables (ELF, PE, Mach-O)
• And 90+ additional formats

Procedure

Data augmentation: 50% of samples use random byte offset to reduce position bias.

Evaluation Results

Perplexity by Region

Fill-Mask Accuracy

Representation Quality

Architecture: Absolute vs Rotary Position Embeddings

This model uses absolute position embeddings, where each position (0-511) has a learned embedding vector added to the token embedding. This is the original BERT approach.

An alternative is Rotary Position Embeddings (RoPE), used by the RoFormer variant. RoPE encodes relative position through rotation matrices applied to query and key vectors in attention, rather than learning absolute position vectors.

Key finding from our experiments: Both approaches show similar position bias (~47-48% accuracy drop at offset 1000). Position bias is primarily a data distribution issue (files naturally start at offset 0) rather than an architecture limitation.

Magic-BERT achieves slightly better perplexity and probing accuracy, while RoFormer achieves better fill-mask accuracy with fewer parameters.

MLM vs Classification: Two-Phase Training

This is the Phase 1 (MLM) model. The training pipeline has two phases:

When to use each:

• Use this model (MLM) for: fill-mask tasks, research, or as a base for custom fine-tuning
• Use classification model for: file type detection, similarity search, production classification

How to Use

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-mlm", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-mlm")

model.eval()

# Read a binary file
with open("example.pdf", "rb") as f:
    data = f.read(512)

# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")

# Tokenize and mask a position
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
mask_pos = 1  # Mask second token (first is [CLS])
inputs["input_ids"][0, mask_pos] = tokenizer.mask_token_id

# Predict masked token
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits[0, mask_pos].topk(5)

print("Top-5 predictions:", tokenizer.convert_ids_to_tokens(predictions.indices))

Getting Embeddings

from transformers import AutoModel

# Load base model for embeddings
model = AutoModel.from_pretrained(
    "mjbommar/magic-bert-50m-mlm",
    trust_remote_code=True
)

with torch.no_grad():
    outputs = model(**inputs)
    # CLS token embedding
    cls_embedding = outputs.last_hidden_state[:, 0, :]  # [batch_size, 512]
    # Mean pooling
    mean_embedding = outputs.last_hidden_state.mean(dim=1)  # [batch_size, 512]

Limitations

1. Position bias: The model performs best when file content starts at position 0. Accuracy drops ~48% when content starts at offset 1000. This reflects training data distribution, not architectural limitations.

2. Sequence length: Limited to 512 tokens. Longer files require truncation or chunking.

3. Text files: Lower performance on high-entropy or highly variable content (e.g., encrypted data, random bytes).

4. Domain specificity: Trained on common file formats; may not generalize to rare or proprietary formats.

Model Selection Guide

Related Models

• magic-bert-50m-classification: Same architecture fine-tuned for classification (89.7% accuracy)
• magic-bert-50m-roformer-mlm: RoFormer variant with rotary position embeddings
• magic-bert-50m-roformer-classification: RoFormer variant fine-tuned for classification (93.7% accuracy, recommended for production)

Related Work

This model builds on the Binary BPE tokenization approach:

• Binary BPE Paper: Bommarito (2025) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
• Binary BPE Tokenizers: Pre-trained tokenizers for executables are available at mjbommar/binary-tokenizer-001-64k.

Key difference: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.

Citation

A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.

@article{bommarito2025binarybpe,
  title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
  author={Bommarito, Michael J., II},
  journal={arXiv preprint arXiv:2511.17573},
  year={2025}
}