Qwen3-Embedding-4B-NVFP4A16

Qwen/Qwen3-Embedding-4B quantized to NVIDIA NVFP4 weights with BF16 activations (W4A16). This checkpoint is packaged for the compressed-tensors loader and was runtime-verified with the vLLM pooling runner on NVIDIA Blackwell.

Quantization

The emitted recipe uses NVFP4A16 for Linear layers and excludes lm_head:

QuantizationModifier:
  targets: [Linear]
  ignore: [lm_head]
  scheme: NVFP4A16

The serialized compressed-tensors configuration records 4-bit float weights, group_size=16, symmetric tensor-group quantization, and FP8 E4M3 scales. Input and output activation quantizers are null, so activations remain BF16.

The included one-shot log reports a data-free quantization pipeline. Exact historical command-line and input-byte provenance were not pinned, and the artifact predates the later English calibration-file preparation. Accordingly, this checkpoint is not described as English-calibrated.

Official MTEB subset evaluation

Official MTEB 2.18.13 evaluators, five complete English test tasks (ArguAna, ArXivHierarchicalClusteringP2P, Banking77Classification.v2, BIOSSES, CQADupstackGamingRetrieval) — a targeted subset, not the full MTEB English v2 suite. Evaluated against the BF16 original and the W4A4 sibling, at every Matryoshka width used in production, same protocol for all three.

TL;DR: ~2.7GB weights (vs ~8GB BF16, ~3x smaller) at ~99% of BF16 MTEB score, consistent from 768 to 2048 dims. The W4A4 sibling trades a bit more quality (~97-98%) for faster FP4 activations.

Average score across the 5 tasks:

Per-task scores at each dimension

Retrieval tasks (ArguAna, CQADupstackGamingRetrieval) show the largest quantization gap (~2-3% relative); clustering and classification are within noise of BF16 at every width. Raw embedding cosine similarity vs BF16 (separate 100-text check, not a task score): mean ~0.95-0.96 across dims.

Matched protocol

• Official MTEB task evaluators, complete test splits, dataset revisions pinned in the uploaded manifests.
• 64,880 encoded texts per arm; 24 concurrent request workers, batch size 32.
• Query instruction applied only to query batches; passages/documents were bare.
• Inputs longer than 8,192 tokens were right-truncated identically across all three arms.
• MTEB cache disabled, isolated per-arm prediction artifacts, no result exceptions.

Raw scores: benchmarks/mteb_dims_comparison/all_scores.json.

Serving with vLLM

vllm serve ironbcc/Qwen3-Embedding-4B-NVFP4A16 \
  --served-model-name qwen3-embedding-4b \
  --runner pooling --convert embed \
  --pooler-config '{"dimensions": 768}' \
  --max-model-len 8192

The tested server returned finite, unit-normalized 768-dimensional vectors and accepted long inputs with explicit endpoint-side truncation.

Queries should carry the base model's retrieval instruction; documents should remain bare:

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query: {your query}

OpenAI-compatible example:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
query = (
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\n"
    "Query: What is NVFP4?"
)
response = client.embeddings.create(
    model="qwen3-embedding-4b",
    input=[query],
    extra_body={"truncate_prompt_tokens": 8192, "truncation_side": "right"},
)
embedding = response.data[0].embedding

Intended use

• Blackwell-oriented, memory-efficient text embeddings served through vLLM.
• Retrieval, clustering, classification, and semantic-similarity workloads after validating the checkpoint on the target domain.
• Query/document asymmetric retrieval using the Qwen3 instruction format.

Limitations

• Evaluation covers five English MTEB tasks, not the full MTEB English v2 or multilingual suites. The table must not be read as a leaderboard aggregate.
• BF16 baseline is now included (see above): W4A16 retains ~99% of BF16 average score across the tested widths, versus ~97-98% for W4A4. This does not isolate activation precision as the sole cause of the remaining W4A16/W4A4 gap.
• Runtime verification here is for vLLM pooling on Blackwell. Direct SentenceTransformers/Transformers loading and other GPU architectures were not revalidated after quantization.
• FP4 kernels require a recent compatible vLLM/FlashInfer stack. The tested environment needed FLASHINFER_DISABLE_VERSION_CHECK=1 because its FlashInfer Python/cubin package versions did not match.
• Inputs were capped at 8,192 tokens for the reported benchmark.
• GPU-memory observations were host-wide point samples with other GPU0 services present; they are not isolated model-allocation measurements.
• Inherits the base model's license, limitations, and biases.

Provenance and checksums

• config.json SHA-256: 0d66a32f634ab1196ff724f536a6f4af1a6c81108470d82eaf05445fb2c3be84
• model.safetensors SHA-256: 2ea69117761426f841907430ec03c8aa979fd12ca6c2bebfb805436554350ce4
• recipe.yaml SHA-256: 051768d850068590f7ac0c0d10441378f8c759e9db444bd228b0fb1ee8c5dcb0

recipe.yaml and oneshot_2026-08-05_03-26-07.log are included as emitted build artifacts. The exact historical build invocation remains unpinned.