DeepGlint-AI/mlcd-vit-large-patch14-336

[Paper] [GitHub]

Model

We used the same Vision Transformer architecture ViT-L/14@336px as CLIP.

Data

Our model was trained on publicly available image-caption data from the LAION400M and COYO700M datasets.

Performance and Limitations

A. MLLMs Evaluation Results

In our experiments, we replaced the CLIP model in LLaVA-NeXT with the MLCD model to demonstrate the performance of the MLCD model in Multimodal Large Language Models (MLLMs). For the language model, we used Qwen2.5-7B. The evaluation results show that the modified model performs exceptionally well across multiple benchmarks, validating the effectiveness of the MLCD model within MLLMs.

B. Linear Probe Evaluation Results

This table presents the results of linear probe evaluations comparing CLIP and MLCD models on the ViT_L_14_336px architecture across various datasets. The linear probe test freezes the pre-trained model's weights and trains a linear classifier on top to assess how well the model's representations generalize to different tasks.

C. Limitations

Models with higher resolution are more friendly to OCR results. We are currently training such models and will soon make them available.

Acknowledgments

We would like to express our gratitude to Xie Yin and Yumeng Wang for their significant contributions to the experimental validation in MLLMs.