中文Sign up

Ruggero1912/Patch-ioner_talk2dino_capdec_COCO_Captions

🤗 Hugging Face sourceimage-to-textapache-2.0273 MBother✓ 1 checksumupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo Ruggero1912/Patch-ioner_talk2dino_capdec_COCO_Captions ./model-folder
Needs a seeder →

Patch-ioner_talk2dino_capdec_COCO_Captions

This repository contains a pre-trained DECAP model from the Patch-ioner framework, presented in the paper "One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework". Patch-ioner is designed for dense image captioning and controllable visual description.

🎯 Model Overview

  • Model Type: DECAP
  • Configuration: mlp_noise.k.yaml
  • Vision Backbone: dinov2_vitb14_reg
  • Language Model: GPT-2
  • Input Resolution: 518x518
  • Prefix Size: 768

DeCap Configuration

  • Memory Bank: Not used
  • Projection Type: coco
  • Linear Talk2DINO: False

📊 Performance

Task METEOR CIDEr SPICE
Image Captioning 0.215 0.655 0.155
Narratives 11.500 29.300 12.300

📈 Detailed Results

Image Captioning Results

  • METEOR: 0.2147
  • CIDEr: 0.6553
  • SPICE: 0.1551
  • BLEU_4: 0.1957
  • ROUGE_L: 0.4542
  • CLIP-S: 0.7090

Narratives Results

  • METEOR: 11.5000
  • CIDEr: 29.3000
  • SPICE: 12.3000
  • BLEU_4: 3.0000
  • ROUGE_L: 24.7000
  • CLIP-S: 66.7000

🚀 Quick Start (Patch-ioner library)

from patch_ioner import load_model, Patchioner

# Load the model
config_path = "config.yaml"
model = load_model(config_path)

# Run inference
image_path = "your_image.jpg"
results = model.forward(image_path)
print(results)

🚀 Quick Start (Transformers library)

from transformers import AutoModel

MODEL_ID = "Ruggero1912/Patch-ioner_talk2dino_capdec_COCO_Captions" # Example model ID

model = AutoModel.from_pretrained(MODEL_ID, trust_remote_code=True)
# Further usage would follow the Hugging Face Transformers pattern

📁 Repository Contents

  • config.yaml: Model configuration file
  • coco_karpathy-009.pt: Pre-trained model weights
  • README.md: This file

🔧 Installation

pip install git+https://github.com/Ruggero1912/Patch-ioner

💡 Usage Examples

Refer to the Patch-ioner repository for updated usage examples.

🎛️ Model Configuration

  • Prefix Size: 768
  • Memory Bank Size: 0
  • Normalization: True

📈 Training Details

  • Training Dataset: COCO Captions
  • Training Epochs: TBD
  • Batch Size: TBD
  • Learning Rate: TBD
  • Optimizer: AdamW

📚 Citation

If you use this model in your research, please cite our paper, refer to the Project Page for updated citation template.

🤝 Contributing

We welcome contributions to improve the Patch-ioner framework. Please see the main repository for contribution guidelines.

📄 License

See the main repository for detailed license information.

🐛 Issues and Support

For issues related to this model or the Patch-ioner framework, please:

  1. Check the main repository for existing issues
  2. Open a new issue with detailed information about your problem
  3. Contact the authors.

🔗 Related Models

Explore other Patch-ioner model configurations:

More models available in Ruggero1912's models

This model is part of the Patch-ioner framework for dense image captioning and controllable visual description.