Qwen3-Reranker-4B-W8A8

INT8 weight + INT8 activation (W8A8) quantization of Qwen/Qwen3-Reranker-4B in the compressed-tensors format for vLLM.

Why: an 8-bit checkpoint that runs on both Ampere (RTX 3090) and Turing (RTX 2080 Ti). Marlin and FP8 kernels need sm80+/sm89+, while vLLM's CUTLASS INT8 W8A8 kernels work from sm75.

Quantization

• llm-compressor 0.14.0: SmoothQuant (strength 0.8) + GPTQ, scheme W8A8: weights int8 symmetric per-channel (static), activations int8 symmetric per-token (dynamic).
• lm_head is not quantized — vLLM builds the classifier from its yes/no rows. The last 4 decoder layers (32–35) are kept in bf16 (max |ΔP(yes)| to bf16: 0.022 → 0.008).
• Calibration: 256 samples, max 1024 tokens, formatted with the original reranker prompt — Russian-language texts of a services marketplace, half relevant (title → own document) and half random query–document pairs.

Check against bf16 (held-out pairs, not used for calibration)

The numbers come from the quantized model in transformers (simulated quantization) right after calibration.

Usage (vLLM)

vllm serve megafetis/Qwen3-Reranker-4B-W8A8 --runner pooling --served-model-name rr4 \
  --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \
  --chat-template qwen3_reranker.jinja --max-model-len 8192 --enable-prefix-caching
# RTX 2080 Ti (no bf16): add --dtype half

qwen3_reranker.jinja — from vLLM examples/pooling/score/template/. Without the template scores are meaningless. Endpoint — /v1/rerank (Jina format) or /score.

Recipe — recipe.yaml; the scripts — ai-models/ of the project that produced it. License — Apache 2.0, as the base model.