leanvocab-bpe-16k

A 6-layer, 384-wide BERT-shaped encoder pretrained from scratch with masked language modelling on 2.00 GB of FineWeb-Edu text, using a 16,000-token vocabulary (Byte-level BPE).

This is the small-vocab candidate arm of leanvocab, a controlled study asking whether a tokenizer with an 8k-16k vocabulary can reach sentence-embedding quality within 0.5 MTEB-avg points of a same-recipe MiniLM-shaped encoder using BERT's 30,522-token vocabulary, while freeing 25-38% of the parameters and without slower end-to-end inference.

This is a research checkpoint, not a finished embedding model. It has only been pretrained with MLM on a small budget; the contrastive fine-tuning that would make it produce useful sentence embeddings has not run yet. See Intended use.

Summary

Bits per byte is the held-out MLM loss per byte of raw text rather than per token. That is the point: two tokenizers cut the same sentence into different numbers of tokens, so per-token loss is not comparable across them, while per-byte loss is. Lower is better.

Why this exists

An encoder of this shape spends a large share of its parameters on the token embedding matrix - 22.47M total for the 30,522-token control. If a much smaller vocabulary costs little quality, that block shrinks and the savings are free. To make the tokenizer the only moving part, every arm shares the corpus, the bytes of text per optimizer step, the model shape, the optimizer, the schedule, the masking and the seed.

All finished arms (identical recipe, seed 0)

Reading of these numbers, as of the runs above:

• bpe-16k (16,000-token small-vocab candidate) beat the control by 0.032 bits/byte while using 25% fewer encoder parameters and 29% less wall clock.
• bpe-8k (8,000-token small-vocab candidate) landed within 0.006 bits/byte of the control for 38% fewer encoder parameters - close enough that a single seed cannot separate them.
• factorized-bert-orig (30,522-token factorized-embedding baseline) came out 0.654 bits/byte worse than the control despite 35% fewer encoder parameters, so its parameter saving did not come for free under this recipe.
• Parameter savings are not interchangeable: an equally sized saving from embedding factorization behaves very differently from one from a smaller vocabulary, as the table shows.

Intended use and limitations

Use it for: reproducing or extending the study, comparing tokenizers under a fixed compute budget, or as a starting point for your own fine-tuning experiments on Apple silicon.

Do not use it for: anything in production, and do not read these numbers as a verdict on the study's claim. Specifically:

• Single seed. Every arm ran once, at seed 0. Differences smaller than a few hundredths of a bit per byte are not yet distinguishable from noise.
• Small budget. 2 GB of text and one pass is far short of a production pretraining run; MLM accuracy near 38% reflects that, and arm ordering can change with scale.
• Bits per byte is not MTEB. The study's claim is about sentence-embedding quality. That requires the contrastive fine-tuning and MTEB phases, which have not run. MTEB is deliberately untouched so far - all decisions were made on a held-out dev split.
• No pooler and no sentence-level objective. Mean-pooling these hidden states will give you something, but nothing here was trained to make it good.
• English, educational web text. FineWeb-Edu only, uncased. Expect poor behaviour on other languages, on cased distinctions, and on domains unlike educational web pages. The corpus is filtered web text and carries its biases into this checkpoint.

Training recipe

• Corpus: FineWeb-Edu sample-10BT at a pinned revision, NFC-normalized, exactly deduplicated (BLAKE2b-128), with a frozen held-out split removed before training.
• Budget: 2 GB of text per arm, a single pass, sequence length 256 with RoBERTa-style full-sentence packing.
• Equal-bytes batching: every step consumes about 307.3 kB of raw text rather than a fixed number of tokens, so arms are not accidentally given different amounts of text. For this arm that is 256 sequences in 2 micro-batches.
• Objective: MLM at 15% masking, 80/10/10 mask/random/keep, with a fixed 38 predictions per sequence so the compiled graph has a static shape.
• Shape: 6 layers, hidden 384, 12 heads, FFN 1536, 256 positions, embeddings tied to the MLM output layer.
• Optimizer: AdamW, lr 0.0005, betas (0.9, 0.98), weight decay 0.01 (not on biases or LayerNorm), gradient clipping 1.0, linear warmup over 6% of steps then linear decay.
• Evaluation: held-out bits per byte on 10 MB with masks fixed by seed 1234, every 500 steps.
• Hardware: one Apple M5 Pro (48 GB) via MLX; no CUDA anywhere in the study. Peak GPU memory 24.2 GB.

Files

Optimizer state is not published (it is twice the size of the weights); resuming training needs the local checkpoint directory from the study.

Usage

The weights were trained in MLX but are stored with BertForMaskedLM parameter names, so transformers loads them directly:

from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained("DhruvParmar051/leanvocab-bpe-16k")
tok = AutoTokenizer.from_pretrained("DhruvParmar051/leanvocab-bpe-16k")

inputs = tok("photosynthesis happens in the[MASK] of plant cells.", return_tensors="pt")
logits = model(**inputs).logits

There is no pooler in the checkpoint (the MLM recipe never trains one), so a BertModel load will report a randomly initialized pooler - that is expected.

One tokenizer detail worth knowing: this vocabulary uses a byte-level pre-tokenizer, so a space written before [MASK] becomes its own Ġ token - a shape the packed pretraining sequences never contained, and predictions at that position get noticeably worse. Attach the mask to the preceding word (the[MASK], not the [MASK]); the token it predicts carries the leading space itself.

Related repositories

• All 12 tokenizers from the study: DhruvParmar051/leanvocab-tokenizers
• The bert-orig arm (control): DhruvParmar051/leanvocab-bert-orig
• The bpe-8k arm (small-vocab candidate): DhruvParmar051/leanvocab-bpe-8k
• The factorized-bert-orig arm (factorized-embedding baseline): DhruvParmar051/leanvocab-factorized-bert-orig

Provenance

• Run 20260918T062350Z-1e4505cf-bpe-16k, seed 0, config SHA-256 1e4505cf5866, commit e46a9184ca06 (working tree dirty).
• macOS-26.6.2-arm64-arm-64bit, Python 3.11.15, MLX 0.32.2, tokenizers 0.23.2.
• MLX_ENABLE_TF32=1: MLX uses TF32 matmuls on Apple GPUs by default (about 1e-3 relative error, about 1.9x faster). The study records this rather than silently depending on it.
• Tokenizer tokenizer.json SHA-256 25099cdd24c4; round-trip check on 2,000 held-out documents: exact, passed.
• Code, configs and the recorded metrics: https://github.com/DhruvParmar051/leanvocab.

Citation

@misc{leanvocab,
  title  = {leanvocab: a controlled study of small-vocabulary English tokenizers
            for sentence-embedding encoders},
  author = {Parmar, Dhruv},
  year   = {2026},
  url    = {https://github.com/DhruvParmar051/leanvocab}
}

Card generated 2026-11-12 from the recorded run metrics.