DeepGlint-AI/llava-mlcd-qwen2.5-7b

[Paper] [GitHub]

Model

We used MLCD as the Vision Encoder in LLaVA-Next.

Data

Our model was trained on publicly available data from the LLaVA-Pretrain and LLaVA-NeXT-Data datasets.

How to eval

pip install lmms-eval==0.2.0

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python -m accelerate.commands.launch \
  --main_process_port=12581 \
  --num_processes=8 \
  -m lmms_eval \
  --model llava \
  --model_args pretrained=DeepGlint-AI/llava-mlcd-qwen2.5-7b,conv_template=qwen_1_5 \
  --tasks mmbench,mme,mmmu,ocrbench,scienceqa,scienceqa_img,seedbench,gqa,pope,textvqa_val,ai2d,chartqa,docvqa_val,infovqa_val,mmstar \
  --batch_size 1 \
  --log_samples \
  --log_samples_suffix mlcd_llava_qwen2_7b \
  --output_path ./log

Performance and Limitations

In our experiments, we replaced the CLIP model in LLaVA-NeXT with the MLCD model to demonstrate the performance of the MLCD model in Multimodal Large Language Models (MLLMs). For the language model, we used Qwen2.5-7B. The evaluation results show that the modified model performs exceptionally well across multiple benchmarks, validating the effectiveness of the MLCD model within MLLMs.

C. Limitations

Models with larger datasets will perform better on more tasks. We are currently training such models and will soon make them available.

Acknowledgments

We would like to express our gratitude to Yumeng Wang for his significant contributions to the experimental validation in MLLMs.