Qwen3.8-9B-Distill (W4A16 Quantized via AutoRound)

This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of empero-ai/Qwen3.8-9B-Distill, quantized using Intel's AutoRound algorithm.

---

⚡ Quantization Details

Calibrated and quantized with fine-grained group sizes and high iteration depth to preserve reasoning traces (<think> blocks) and multimodal capabilities:

• Algorithm: Intel AutoRound
• Precision / Scheme: W4A16 (4-bit weights, 16-bit activations)
• Group Size: 32 (fine-grained reconstruction fidelity)
• Symmetric (sym): True
• Calibration Samples (nsamples): 512
• Sequence Length (seqlen): 4096
• Tuning Iterations (iters): 1000 (Production-grade accuracy)
• Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
• Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

---

📦 Available Formats

Depending on your inference engine, choose the appropriate repository:

• AutoRound Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound
• AutoGPTQ Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-GPTQ
• LLM-Compressor / Compressed-Tensors Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-LLM-Compressor

---

🚀 Usage & Quickstart

1. High-Throughput Serving via vLLM

# Using the LLM-Compressor / Compressed-Tensors build
vllm serve Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-GPTQ \
    --dtype bfloat16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

---

📊 VRAM & Performance Benefits

• Original Model (BF16): ~8–10 GB VRAM required for full context inference
• Quantized Model (W4A16 Group 32): ~2.5–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs, laptops, and edge devices)
• Throughput: Lowers memory bandwidth pressure, accelerating token generation speeds during extended chain-of-thought (<think>) reasoning.

---

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.15

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---

📚 Acknowledgments

• Original Distilled Model: Developed by Empero AI
• Base Architecture: Qwen Team (Alibaba)
• Quantization Framework: Intel AutoRound