distilvit

This model is a work in progress. Fine-tuned version of those base models:

• a VIT model for the image encoder: https://huggingface.co/google/vit-base-patch16-224-in21k
• a Distilled GPT-2 model for the text decoder: https://huggingface.co/distilbert/distilgpt2

This model was trained on:

• Flickr30k : https://huggingface.co/datasets/nlphuji/flickr30k
• COCO 2017: https://cocodataset.org

You can get that checkpoint using the 3083a3cef6e3c8dd90df3f088074bbe836b0f403 commit.

It was then further fine-tuned on :

• Flickr30k debiased
• DocOrNot
• Alt Text Validation

For the latter, the dataset was annotated by our team to correct the alt text generated by the model, using the checkvite tool.

You can find the code used to create the model here: https://github.com/mozilla/distilvit