UltraBERT-Embed-150M

UltraBERT-Embed-150M is an English text embedding model built on UltraBERT-150M. It maps text to a 768-dimensional vector, takes inputs up to 32K tokens, follows task instructions on the query side, and embeds code as well as text. On MTEB(eng, v2) it scores 67.24 Mean(Task), the highest of any dense model under 200M parameters, without distillation from a larger embedding model.

For more information about UltraBERT, please check out our paper.

✨ Highlights

As of October 11, 2026, UltraBERT-Embed-150M is the state of the art among dense embedding models under 200M parameters on both MTEB(eng, v2) and MTEB(Code, v1).

• MTEB(eng, v2): 67.24. Mean over the 41 tasks, ahead of Tarka-Embedding-150M-V1 (66.39) and gte-modernbert-base (66.26). The models closest behind are distilled from larger embedding models; this one is trained with contrastive losses on public pairs only.
• MTEB(Code, v1): 73.30. Mean over the 12 tasks, with CoIR 72.73, from code pairs in both training stages.
• LongEmbed: 79.30. Highest of the models we compared, including 99.0 on passkey retrieval, with documents up to 32K tokens embedded whole rather than truncated.
• RAR-b: 28.75. Highest of the encoders we compared on reasoning-style retrieval.
• Instructions. Queries carry an E5-mistral style instruction (Instruct: {task}\nQuery: ) and documents are embedded raw, so one model serves retrieval, classification, clustering and similarity.

🚀 Usage

Requires sentence-transformers >= 6.0 and transformers >= 5.20; on earlier 5.x versions, pass trust_remote_code=True. FlashAttention 2 is recommended: with it, Sentence Transformers packs the batch without padding, which matters for long inputs. SDPA also works ("attn_implementation": "sdpa").

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "dmis-lab/UltraBERT-Embed-150M",
    model_kwargs={"attn_implementation": "flash_attention_2", "dtype": "bfloat16"},
)

queries = ["Win big, click now!"]
labels = ["spam", "promotional", "announcement"]

query_embeddings = model.encode(queries, prompt="Instruct: Labeling spam emails\nQuery: ")
label_embeddings = model.encode(labels, prompt="Label: ")
print(model.similarity(query_embeddings, label_embeddings))

prompt_name="query" applies the default retrieval instruction. Documents never take an instruction, except label phrases, which take the prefix Label: . For symmetric tasks (STS, paraphrase, duplicate detection) apply the same instruction to both sides, e.g. Instruct: Retrieve semantically similar text.\nQuery: . For classification and clustering, name the categories, e.g. Instruct: Classify the sentiment expressed in the given movie review text\nQuery: .

📊 Evaluation

MTEB(eng, v2)

MTEB(Code, v1) and CoIR

LongEmbed

RAR-b

🏛️ Model Description

⚙️ Training

UltraBERT-Embed-150M is trained from UltraBERT-150M in two contrastive stages with Sentence Transformers. The first stage trains on about 500M weakly supervised text and code pairs with in-batch negatives only. The second stage fine-tunes on about 5.7M examples with mined hard negatives, with a task instruction on the query side and inputs up to 32K tokens so that long documents are seen whole. The released weights are the average of several checkpoints from the second stage.

Training data comes from public sources only, without distillation from a larger embedding model:

• Web question-answer and title-body pairs (PAQ, GooAQ, Yahoo Answers, Amazon QA, Reddit, StackOverflow, news, Wikipedia, S2ORC)
• Code pairs (CornStack, OCGI, xCodeEval)
• Retrieval datasets with hard negatives (MS MARCO, NQ, TriviaQA, HotpotQA, FEVER, SQuAD 2.0, MIRACL, MLDR, DBpedia, BioASQ, PubMedQA, MedQuAD)
• Sentence similarity (NLI, QQP, PAWS-X, synthetic STS)
• Classification and clustering (synthetic classification data from the Nemotron embedding dataset, Amazon polarity, toxicity datasets)

Citation

@misc{park2026ultrabert,
      title={UltraBERT: Beyond the Limit of BERT Pre-Training}, 
      author={Jungwoo Park and Taewhoo Lee and Jaewoo Kang},
      year={2026},
      eprint={2610.15028},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.15028}, 
}