Qwen3-VL-8B-Instruct-W4A16-AutoRound-GPTQ

Model Overview

This is a 4-bit quantized GPTQ version of the state-of-the-art Qwen/Qwen3-VL-8B-Instruct vision-language model.

Unlike standard GPTQ conversions which rely on a greedy layer-wise algorithm, this model was optimized using Intel's AutoRound. AutoRound analyzes the model's weights over 800 tuning steps with 512 calibration samples to find the optimal quantization points. This results in significantly lower perplexity and better reasoning retention than standard GPTQ, while maintaining full compatibility with all GPTQ inference backends.

Key Highlights

• Best-in-Class Quality: tuned for 800 iterations to preserve the model's complex visual reasoning capabilities.
• Uncompromised Vision: The visual encoder (Vision Tower) is kept in FP16 (Unquantized) to ensure no degradation in OCR, chart reading, or spatial analysis.
• Broad Compatibility: Works with AutoGPTQ, Transformers, and older vLLM versions that support GPTQ.

Technical Specifications

---

Quickstart

1. Installation

To run this model, you need transformers and the auto-gptq kernel library.

pip install auto-gptq transformers torch

2. Inference Example

This snippet demonstrates how to load the model and analyze an image.

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

# 1. Load the Model
model_id = "Vishva007/Qwen3-VL-8B-Instruct-W4A16-AutoRound-GPTQ"

model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

# 2. Load the Processor
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# 3. Prepare Input (Image + Text)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }
]

# 4. Process Inputs
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to(model.device)

# 5. Generate Output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)

print(f"Model Response:\n{output_text}")

---

Performance & Benchmarks

This quantized model aims to match the performance of the FP16 original model while reducing memory usage by nearly 70%.

• VRAM Usage: reduced from ~16GB (FP16) to ~5.5GB (GPTQ).
• Throughput: Higher token generation speed on memory-bandwidth limited GPUs (like RTX 3090, 4090, L40).

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.15

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---

Acknowledgements

• Base Model: Qwen/Qwen3-VL-8B-Instruct
• Quantization Tool: Intel AutoRound
• Paper: Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs

Citation

If you use this model, please cite the original Qwen3-VL paper:

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report}, 
      author={Qwen Team},
      year={2025}
}