vminibert-108M

vminibert-108M is a 108.6M-parameter Transformer encoder trained from scratch as a Masked Language Model on a blended 4-domain corpus (WikiText-103, BookCorpusOpen, FineWeb, and C4) using dynamic Whole-Word Masking (WWM).

Rather than a stock BERT reimplementation, it uses a modern LLaMA-style encoder recipe:

• RoPE (Rotary Position Embeddings) instead of learned absolute positions
• RMSNorm instead of LayerNorm
• GeGLU feed-forward blocks instead of a plain GELU MLP

Model Details

Training Data

Trained on ~250k samples blended from four domains:

Masking used dynamic Whole-Word Masking at a 15% rate.

Evaluation

Evaluated on official unseen test splits with standard 15% WWM:

MRR@10 for wikitext-2 is not yet computed — fill in or remove the column before publishing.

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

model_id = "vprojectx/VMiniBert"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id, trust_remote_code=True)

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
results = fill_mask("The capital of France is [MASK].")

for res in results:
    print(f"{res['token_str']:<12} {res['score']:.4f}")

Limitations & Intended Use

• Context window: pretrained and evaluated at 32-token sequences only; behavior beyond that is untested. This is the single biggest caveat for anyone evaluating this model — flag it up front, not just in a bullet list.
• Corpus size: ~250k samples is modest for a from-scratch LM; expect it to need task-specific fine-tuning for downstream use rather than strong zero-shot performance.
• Intended use: a lightweight, fast MLM encoder for fine-tuning (classification, NER, masked-token extraction) and as a reference implementation of a modern (RoPE/RMSNorm/GeGLU) BERT-style architecture trained end-to-end from scratch.
• Not intended for: long-document tasks, production deployment without fine-tuning, or use cases requiring calibrated confidence at scale (no calibration study was done).

Citation

If you use this model, please cite it as:

@misc{vminibert2026,
  author = {vprojectx},
  title = {vminibert-108M: A From-Scratch BERT-Style Encoder with RoPE, RMSNorm, and GeGLU},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/vprojectx/VMiniBert}
}