potion-mxbai-512d

A static embedding model that outperforms potion-base-32M — the previous best static embedding model — by using a stronger teacher model and matching its architecture. Trained on 1M C4 sentences.

Update: A newer version trained on 2M sentences is available: potion-mxbai-2m-512d — scoring 71.28 avg (+0.55 over this model, +2.23 STS).

Highlights

• 70.73 avg on MTEB English (STS + Classification + PairClassification) vs potion-base-32M's 69.96
• +3.62 STS points over potion-base-32M (69.36 vs 65.74)
• 500x faster than transformer-based embedding models on CPU
• ~32MB model size (63K vocab x 512 dims, float16)
• Pure numpy inference — no GPU needed

How It Was Made

1. Teacher: mixedbread-ai/mxbai-embed-large-v1 (335M params, BERT-large architecture, MTEB 64.68)
2. Custom vocabulary: 56K tokens built from 1M C4 English sentences via corpus frequency analysis
3. Distillation: model2vec distillation with 512-dim PCA
4. Tokenlearn pre-training: Contrastive loss training on 1M C4 sentences using tokenlearn

Benchmark Results

Per-task breakdown vs potion-base-32M

Wins on 11/25 tasks.

Usage

from model2vec import StaticModel

model = StaticModel.from_pretrained("blobbybob/potion-mxbai-512d")
embeddings = model.encode(["Hello world", "Static embeddings are fast"])

With Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("blobbybob/potion-mxbai-512d")
embeddings = model.encode(["Hello world", "Static embeddings are fast"])

Training Details

• Compute: Modal A10G GPU
• Featurization: ~2 hours (1M C4 sentences through mxbai-embed-large-v1)
• Training: ~24 minutes (7 epochs, contrastive loss, early stopping)
• Total cost: ~$5-6 on Modal

Citation

@article{minishlab2024model2vec,
  author = {Tulkens, Stephan and {van Dongen}, Thomas},
  title = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year = {2024},
  url = {https://github.com/MinishLab/model2vec}
}