Qwen3-Embedding-8B-GPTQ (4-bit Quantized)

Overview

qwen3-embed-8b-gptq is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.safetensors (1 file)
• config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja
• Additional configuration, tokenizer, processor, or shard files (10 visible artifacts total)

Quick start

vLLM (documented configuration)

CUDA_VISIBLE_DEVICES=0 vllm serve groxaxo/qwen3-embed-8b-gptq \
  --task embed \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.5 \
  --dtype float16 \
  --quantization gptq

This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

GPTQ W4A16 quantized version of Qwen3-Embedding-8B

• Original Size: 15GB → Quantized Size: 4.5GB (70% reduction)
• Semantic Accuracy: 99.14% preserved
• Context Window: Up to 32k tokens
• Quantization: GPTQ W4A16 (4-bit weights, 16-bit activations)

Model Details

• Base Model: Qwen/Qwen3-Embedding-8B
• Parameters: 8B
• Embedding Dim: 3072
• Max Sequence Length: 32768
• Quantization: GPTQ (4-bit weights, 16-bit activations)
• Calibration Samples: 128
• Dataset: ultrachat-200k

Performance

Test Results

Comprehensive testing with 11 test pairs:

• Mean Absolute Difference: 0.007 (0.86%)
• Maximum Difference: 0.019 (1.9%)
• Rating: ✅ EXCELLENT (>95% preserved)

Usage

vLLM (Recommended)

CUDA_VISIBLE_DEVICES=0 vllm serve groxaxo/qwen3-embed-8b-gptq \
  --task embed \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.5 \
  --dtype float16 \
  --quantization gptq

Python

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "groxaxo/qwen3-embed-8b-gptq",
    torch_dtype="float16",
    trust_remote_code=True,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "groxaxo/qwen3-embed-8b-gptq",
    trust_remote_code=True
)

# Generate embeddings
inputs = tokenizer("Your text here", return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model(**inputs)
embeddings = outputs.last_hidden_state.mean(dim=1)

OpenAI-compatible API

curl http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "groxaxo/qwen3-embed-8b-gptq",
    "input": "Your text here"
  }'

Quantization Details

Method: GPTQ (Group-wise Post-Training Quantization)

Configuration:

• Weights: 4-bit
• Activations: 16-bit
• Group Size: 128
• Scheme: W4A16
• Damping Factor: Default (auto)

Why GPTQ over AWQ?

• AWQ calibration produced NaN values for Qwen3-Embedding-8B
• GPTQ more stable for embedding models
• GPTQ provides better accuracy for this architecture

Benchmark Results

Inference Speed: ~1.2-1.5x slower than FP16 Memory Usage: ~30% of FP16 model Batch Size: 512 (recommended for vLLM) Throughput: Excellent for batch processing

Recommendations

Use this model when:

• ✅ Semantic accuracy is critical (>95% required)
• ✅ Want 70% memory reduction
• ✅ Production deployment with vLLM
• ✅ Batch processing of embeddings

Consider alternatives when:

• ⚠️ Need <25% memory (use Int8)
• ⚠️ Maximum inference speed (use FP16)
• ⚠️ Edge devices with limited compute

Evaluation

Comprehensive evaluation report available in the repository or QUANTIZATION_REPORT.md.

Test Suite:

• 11 diverse test pairs (similar, different, related concepts)
• Cosine similarity measurements
• Statistical analysis of degradation

Limitations

1. Quantization Error: Small accuracy loss (~0.86%)
2. Inference Speed: ~20-50% slower than FP16
3. Not Compatible: With AWQ-optimized servers (use GPTQ)
4. Architecture: Specific to Qwen3-Embedding-8B

Citation

If you use this model, please cite:

@article{qwen2024qwen,
  title={Qwen Technical Report},
  author={Qwen Team},
  year={2024}
}

License

Apache 2.0

Acknowledgments

• Original Model: Alibaba Qwen Team
• Quantization Tool: vLLM/llmcompressor
• Testing: Comprehensive semantic similarity evaluation

---

Model Card Version: 1.0
Last Updated: February 23, 2026