UME-Recipes: What Really Works for Universal Multimodal Embeddings?

Paper · Code and Recipes · Alibaba Multimodal & Industrial AI Team

This repository provides the released models for What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies.

We systematically study representative contrastive training strategies under aligned backbones, training data, optimization budgets, and MMEB evaluation.

---

Key findings

• Controlled comparison: Many sophisticated strategies provide only marginal gains over a properly aligned VLM2Vec baseline, and some degrade overall performance.
• Gradient distortion: Upweighting high-similarity negatives can hurt downstream generalization.
• Distribution illusion: A wider positive–negative similarity gap does not necessarily yield better retrieval performance.
• Synergistic training: Combining hard-negative mining, false-negative mitigation, and hard-batch construction achieves the strongest result, improving average MMEB performance by 6.0% with Qwen2-VL-2B.

The paper also introduces an MLLM-as-a-Judge diagnostic framework that categorizes top-1 retrieval outcomes into labeled positives, false negatives, and hard or easy negative errors.

---

Released models

We release the best-performing model for each evaluated backbone under our controlled training and evaluation setting. Across Qwen2-VL-2B, Qwen2-VL-7B, and Qwen3.5-2B, the strongest configuration is obtained with the B3 training recipe.

The released artifacts are PEFT DoRA adapters, not standalone copies of their base models. Use them with the corresponding Qwen backbone and the model implementation in UME-Recipes. All models use last-token pooling followed by L2 normalization and were trained with an image pixel range of 4,096–2,257,920.

MMEB results

Scores are mean Precision@1 (%) on the corresponding MMEB datasets. Overall is the average across all 36 image-task datasets.

---

Download

Download one released adapter with the Hugging Face CLI:

hf download alibaba-multimodal-industrial-ai/UME-Recipes \
  --include "UME-B3-Qwen2-VL-2B/*" \
  --local-dir ./released-models

Replace UME-B3-Qwen2-VL-2B with another model directory from the table when needed.

Evaluation

Install the official repository and run its MMEB evaluation scripts:

git clone https://github.com/alibaba-multimodal-industrial-ai/UME-Recipes.git
cd UME-Recipes
python -m pip install -e .

bash scripts/eval_all_qwen2_vl.sh \
  /path/to/Qwen2-VL-2B-Instruct \
  /path/to/released-models/UME-B3-Qwen2-VL-2B \
  ./exp/evaluation/UME-B3-Qwen2-VL-2B

For Qwen3.5, use scripts/eval_all_qwen3_5.sh. See the GitHub repository for environment setup, training recipes, data layout, and complete evaluation instructions.

---

License

The released adapters are provided under the Apache License 2.0. The corresponding base models, datasets, and evaluation resources remain subject to their original licenses and terms.

---

Citation

If you find this work useful, please cite:

@article{li2026what,
  title={What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies},
  author={Li, Duo and Pan, Haojun and Yang, Zuhao and Wang, Xintong and Huo, Chengfu and Lu, Shijian and Ding, Liang},
  journal={arXiv preprint arXiv:2610.15035},
  year={2026}
}