potion-mxbai-2m-512d

A high-quality static embedding model that outperforms potion-base-32M — trained on 2M C4 sentences with tokenlearn contrastive pre-training.

Highlights

• 72.13 avg on full MTEB English (STS + Classification + PairClassification, 25 tasks, English subsets only)
• 80-88x faster than all-MiniLM-L6-v2 on CPU (~16K vs ~200 sentences/sec)
• ~125MB model size (29K vocab x 512 dims, float32)
• Pure numpy inference — no GPU needed

How It Was Made

1. Teacher: mixedbread-ai/mxbai-embed-large-v1 (335M params, BERT-large architecture)
2. Custom vocabulary: Built from 2M C4 English sentences
3. Distillation: model2vec distillation with 512-dim PCA
4. Tokenlearn pre-training: Contrastive loss training on 2M C4 sentences using tokenlearn

Benchmark Results (Full MTEB English Suite)

Evaluated on 25 tasks (10 STS, 12 Classification, 3 PairClassification), English subsets only.

Model Family

Usage

from model2vec import StaticModel

model = StaticModel.from_pretrained("blobbybob/potion-mxbai-2m-512d")
embeddings = model.encode(["Hello world", "Static embeddings are fast"])

With Sentence Transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("blobbybob/potion-mxbai-2m-512d")
embeddings = model.encode(["Hello world", "Static embeddings are fast"])

Training Details

• Featurization: 2M C4 sentences encoded by mxbai-embed-large-v1
• Training: Tokenlearn contrastive loss, batch size 256
• Total cost: ~$3-4 on Modal

Citation

@article{minishlab2024model2vec,
  author = {Tulkens, Stephan and {van Dongen}, Thomas},
  title = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year = {2024},
  url = {https://github.com/MinishLab/model2vec}
}