Qwen3.8-27B (W4A16 Quantized via AutoRound)

This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of Qwen/Qwen3.8-27B generated using Intel's AutoRound algorithm.

---

⚡ Quantization Details

The model was calibrated and quantized using production-grade settings to minimize accuracy degradation while significantly lowering VRAM requirements:

• Quantization Algorithm: AutoRound
• Bits / Precision: W4A16 (4-bit integer weights, 16-bit activation)
• Group Size: 64 (provides higher reconstruction fidelity than standard 128)
• Symmetric (sym): True
• Calibration Samples (nsamples): 512
• Tuning Iterations (iters): 600
• Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
• Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) layers preserved in native bfloat16.

---

🚀 Usage & Quickstart

1. Inference via vLLM

For high-throughput production serving:

vllm serve Vishva007/Qwen3.8-27B-W4A16-AutoRound \
    --quantization auto-round \
    --dtype bfloat16 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90

(Note: For the GPTQ format repo, you can set --quantization gptq if required by your backend).

---

📊 VRAM & Performance Benefits

• Original Model (BF16): ~54 GB VRAM required (needs 2x A100/A6000 or 80GB VRAM)
• Quantized Model (W4A16 Group 64): ~16–18 GB VRAM (can easily run on a single RTX 3090 / 4090 / A5000 24GB)
• Speedup: Reduced memory bandwidth bottleneck leading to faster decoding token speeds.

---

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.15

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---

📚 Acknowledgments

• Quantization performed using Intel AutoRound.
• Base architecture provided by the Qwen Team.