HuggingFaceTB__SmolVLM-256M-Instruct__gptq_int4_merged
This is a GPTQ (4-bit) quantized version of HuggingFaceTB/SmolVLM-256M-Instruct.
Quantization Details
- Method: GPTQ
- Bits: 4
- Base model: HuggingFaceTB/SmolVLM-256M-Instruct
- Group size: 128
- Quantized modules: text decoder + lm_head
- FP16 modules: vision encoder, connector
- Approach: Extract inner Llama -> GPTQ -> re-attach to full Idefics3
Usage
from transformers import AutoProcessor, AutoModelForImageTextToText
import torch
model = AutoModelForImageTextToText.from_pretrained(
"{REPO_ID}",
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("{REPO_ID}", trust_remote_code=True)
Replace {REPO_ID} with the repo ID of this model.
Original Model
See HuggingFaceTB/SmolVLM-256M-Instruct for the original FP16 model.