Qwen3-Embedding-4B-NVFP4

Qwen/Qwen3-Embedding-4B quantized to NVFP4 (4-bit weights and 4-bit activations) with llm-compressor, for serving on NVIDIA Blackwell (SM120) under vLLM. Built to back a scripture-retrieval application, so it is additionally patched to expose Matryoshka output dimensions and serve 768-dim vectors directly.

Quantization details

QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head"])

• Weights — FP4, group_size=16, strategy=tensor_group, symmetric, FP8 (e4m3) group scales.
• Activations — FP4, group_size=16, dynamic="local" per-group scales with a static global FP8 scale fitted from calibration data. This is why the calibration set matters: W4A4 has a data-dependent activation scale, unlike a weights-only scheme.
• Excluded — lm_head only. The model is dense (no MoE), so no router/gate exclusions apply. lm_head is unused by the pooling runner and is tied to the input embeddings.
• Architecture is Qwen3ForCausalLM, so it loads with AutoModelForCausalLM; the embedding behaviour comes from last-token pooling, not a separate head.

Calibration

512 samples, max_seq_length=512, English only:

Query-shaped samples carry the model's real query prefix (below) and passages do not, so the calibrated activation scales match the asymmetric way the model is used at inference.

Matryoshka patch — read this if you fork the recipe

Upstream Qwen3-Embedding-4B supports MRL truncation but its config.json declares neither is_matryoshka nor matryoshka_dimensions. vLLM refuses the dimensions pooling parameter unless ModelConfig.is_matryoshka is true, so this checkpoint adds:

{ "is_matryoshka": true,
  "matryoshka_dimensions": [128, 256, 512, 768, 1024, 1536, 2048, 2560] }

vLLM truncates after pooling and renormalizes, so a 768-dim response is unit-norm.

Serving with vLLM

vllm serve ironbcc/Qwen3-Embedding-4B-NVFP4 \
  --served-model-name qwen3-embedding-4b \
  --runner pooling --convert embed \
  --pooler-config '{"dimensions": 768}' \
  --max-model-len 8192

--pooler-config '{"dimensions": 768}' makes every response 768-dim, so OpenAI-compatible clients need no dimensions parameter. Drop it to get the native 2560.

Measured on one Blackwell GPU at gpu_memory_utilization=0.09 (~8.8 GB resident): a 64-text batch (1654 prompt tokens) returns in 57-59 ms, about 0.9 ms/text.

Asymmetric use matters

Embed documents bare, and prefix queries with the string from the base model's config_sentence_transformers.json:

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query: {your query}

This is not cosmetic. On a 31k-verse Bible corpus, "How can I be saved?" retrieves Psalm 55:16 / Genesis 49:18 unprefixed versus Acts 15:11 / Romans 10:10 / Titus 3:5 prefixed; "I am so lonely" goes from Job 7:19 to Psalm 25:16 / 102:7 / 88:8.

Official MTEB subset evaluation

Official MTEB 2.18.13 evaluators, five complete English test tasks (ArguAna, ArXivHierarchicalClusteringP2P, Banking77Classification.v2, BIOSSES, CQADupstackGamingRetrieval) — a targeted subset, not the full MTEB English v2 suite. Evaluated against the BF16 original and the W4A16 sibling, at every Matryoshka width used in production, same protocol for all three.

TL;DR: ~2.7GB weights (vs ~8GB BF16, ~3x smaller), weights and activations in FP4, at ~97-98% of BF16 MTEB score from 768 to 2048 dims. The W4A16 sibling trades the activation-speed gain for ~99% BF16 quality instead.

Average score across the 5 tasks:

Per-task scores at each dimension

Retrieval tasks (ArguAna, CQADupstackGamingRetrieval) take the biggest hit under W4A4 (~4-5% relative on CQADupstackGamingRetrieval); clustering and classification stay close to BF16. Raw embedding cosine similarity vs BF16 (separate 100-text check, not a task score): mean ~0.95-0.96 across dims.

Matched protocol

• Official MTEB task evaluators, complete test splits, dataset revisions pinned in the uploaded manifests.
• 64,880 encoded texts per arm; 24 concurrent request workers, batch size 32.
• Retrieval instruction applied only to queries; documents remained bare.
• Inputs longer than 8,192 tokens were right-truncated identically across all three arms.
• MTEB cache disabled, isolated per-arm prediction artifacts, no result exceptions.

Raw scores: benchmarks/mteb_dims_comparison/all_scores.json.

Downstream check on the target application: with W4A4 vectors, dense retrieval over a 31,086-verse corpus returns John.3.16 as top-1 for "For God so loved the world" (cos 0.7865) and Psalm.23.1 for "the Lord is my shepherd" (0.7936).

Limitations

• Reported MTEB results cover five English tasks, not the full English v2 or multilingual suites; do not treat them as a leaderboard aggregate.
• BF16 baseline is now included (see above): W4A4 retains ~97-98% of BF16 average score across the tested widths, versus ~99% for W4A16.
• W4A4 is lossy in a way that shows up directly in embedding space — see above. Validate on your own retrieval set before trusting it in production.
• Blackwell/SM120-oriented. FP4 activation kernels need recent flashinfer; on vLLM 0.26.0 the flashinfer-python==0.6.14 pin has no matching flashinfer-cubin release, which may require FLASHINFER_DISABLE_VERSION_CHECK=1.
• English calibration only. For multilingual retrieval, recalibrate on your languages.
• Inherits the base model's Apache-2.0 license and its biases.

Provenance

Quantized with llm-compressor 0.12.0.1 in an environment isolated from the serving stack (transformers 5.10.1, torch 2.11.0+cu130). Conversion and calibration ran on a single Blackwell GPU. recipe.yaml and the oneshot_*.log in this repo are the actual artifacts emitted by the run.