Qwen3.8-27B (official) — GPTQ-Pro INT4 g64, calib-128

Overview

Qwen3.8-27B-GPTQ-Pro-4bit-g64-calib128 is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.safetensors (5 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• processor_config.json
• chat_template.jinja
• quantize_config.json
• Additional configuration, tokenizer, processor, or shard files (15 visible artifacts total)

Quick start

vLLM (documented configuration)

vllm serve groxaxo/Qwen3.8-27B-GPTQ-Pro-4bit-g64-calib128 \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --dtype bfloat16 --language-model-only \
  --max-model-len 260000 --gpu-memory-utilization 0.92 --max-num-seqs 2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

INT4 GPTQ-Pro quantization of the official Qwen/Qwen3.8-27B, with the MTP (multi-token prediction) draft head preserved and verified working in vLLM speculative decoding.

Sister build (abliterated source): groxaxo/Huihui-Qwen3.8-27B-abliterated-GPTQ-Pro-4bit-g64-calib128 — same recipe, same loss profile.

Quantization details

MTP verification + a critical serving flag

vLLM 0.24.0, TP=2: Qwen3_5MTP drafter resolves and loads; drafts accepted (25/26 on smoke test); coherent generations. Single RTX 3090 cannot fit INT4 27B + MTP drafter — use TP≥2.

⚠ If you serve this on multi-GPU RTX 3090 (PCIe, no NVLink) and get garbage output (e.g. !!!!...), pass --disable-custom-all-reduce. vLLM's custom allreduce kernel silently corrupts tensor-parallel output on this hardware class. Verified empirically: identical checkpoint, identical prompts — broken without the flag, byte-identical correct output with it. This affects any TP-served model on such boxes, not just this quant.

Speed reference from the sister Huihui build (identical recipe/arch, 2× RTX 3090 TP=2, temp 0.7, 256 tok): 78–87 tok/s single-stream with MTP draft-3 vs ~71 no-MTP; draft acceptance ~59–77% depending on temperature.

Usage

vllm serve groxaxo/Qwen3.8-27B-GPTQ-Pro-4bit-g64-calib128 \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --dtype bfloat16 --language-model-only \
  --max-model-len 260000 --gpu-memory-utilization 0.92 --max-num-seqs 2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

min_p and logit_bias do not work with speculative decoding (vLLM limitation).

Acknowledgements

• Qwen — Qwen3.8-27B
• GPTQModel / ModelCloud for GPTQ-Pro