Fine tuned version of xtuner/llava-llama-3-8b-v1_1 on gokaygokay/random_instruct_docci dataset.
pip install git+https://github.com/haotian-liu/LLaVA.git --no-deps
pip install lmdeploy
# Google Colab Error Fix
import nest_asyncio
nest_asyncio.apply()
from lmdeploy import pipeline
from lmdeploy.vl import load_image
pipe = pipeline('gokaygokay/llava-llama3-docci')
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
response = pipe(('describe this image', image))
print(response)
Academic use and citation
This model is evaluated as a dense-captioning baseline in Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding (EMNLP 2024).
If this model is useful in your research, please cite both the model and the paper that documents its academic use:
@software{aydogan2024llava_llama3_docci,
author = {Aydoğan, Gökay},
title = {LLaVA-Llama-3-DOCCI},
year = {2024},
url = {https://huggingface.co/gokaygokay/llava-llama3-docci},
note = {Hugging Face model}
}
@inproceedings{chung2024selective,
author = {Chung, Jiwan and Lee, Sungjae and Kim, Minseo and Han, Seungju and Yousefpour, Ashkan and Hessel, Jack and Yu, Youngjae},
title = {Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding},
booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
year = {2024},
pages = {2423--2451},
doi = {10.18653/v1/2024.emnlp-main.143}
}
Maintainer: Gökay Aydoğan