Qwen3-Embedding-4B GGUF — Quantized by BatiAI

GGUF quantizations of Qwen/Qwen3-Embedding-4B — the mid-tier sweet spot for 16 GB+ Macs. Higher retrieval quality than 0.6B; leaner RAM than 8B. Part of BatiAI's on-device RAG stack for BatiFlow.

Quick Start

Ollama

ollama pull batiai/qwen3-embedding:4b        # Q6_K (3.3 GB, recommended default)
ollama pull batiai/qwen3-embedding:4b-q8     # Q8_0 (4.3 GB, max quality)

curl http://localhost:11434/api/embeddings -d '{
  "model": "batiai/qwen3-embedding:4b",
  "prompt": "semantic search query"
}'

llama.cpp

./llama-server \
  -m Qwen3-Embedding-4B-Q8_0.gguf \
  --embeddings --pooling last -c 32768 \
  --host 127.0.0.1 --port 8080

Available Quantizations

Why Q6 over Q8 as default? Measured drift 0.998 = cosine distance well below any retrieval noise floor. Saving ~1 GB matters on 16 GB Macs where every GB of free RAM helps. If you want maximum conservatism, pull :4b-q8.

Why no IQ3 / IQ4 for embedding? Unlike chat LLMs, embedding quality cascades into cosine-similarity drift at low bit-widths. Q6_K / Q8_0 are the safe range.

Quality Verification (measured)

Four-stage harness run on both quants. Full testset + script reproducible via scripts/bench-embedding-quality.sh.

All stages PASS with comfortable margin. Q8_0 and Q6_K produce essentially identical retrieval quality on this testset, with Q6_K saving ~25 % on disk.

Quality tier comparison (across BatiAI text-embedding lineup)

4B sits in the middle — meaningfully better separation than 0.6B, within striking distance of 8B, at roughly half the disk footprint of 8B. Recommended default for 16 GB+ Macs.

Matryoshka — runtime-configurable dimension

Qwen3-Embedding-4B outputs up to 2560 dimensions. BatiFlow RAG stack defaults to 1024 (quality / latency sweet spot per our tests). Truncate at read time — no re-embed needed:

emb = get_embedding(text)     # [2560]
emb_1024 = emb[:1024]         # truncate for storage savings
import numpy as np
emb_1024 = emb_1024 / np.linalg.norm(emb_1024)   # re-normalize

Why text-only?

Qwen3-Embedding-4B is designed specifically for text. For multimodal (image + text), see Qwen3-VL-Embedding-8B on BatiAI.

RAG Stack Integration

user query
   ↓ [Qwen3-Embedding 4B]            ← YOU ARE HERE (mid tier)
1024-dim vector
   ↓ vector DB (sqlite-vec / LanceDB)
top-K candidates
   ↓ [Qwen3-Reranker 4B or 8B]
top-3
   ↓ [Qwen3.6-35B-A3B chat LLM]
answer

Recommended Usage — query vs document

# Query side
query = "Instruct: Given a document query, retrieve the most relevant chunk.\n" \
        "Query: " + user_input

# Document side — no instruction prefix
document = chunk_text

BatiFlow handles this automatically. For custom integrations, see Qwen3-Embedding usage.

Why BatiAI?

• Quantized directly from Qwen's BF16 safetensors
• general.author: BatiAI for provenance
• 4-stage quality harness (same-lang, cross-lingual, real-doc top-1, quant drift) — numbers published
• Paired with Qwen3-Reranker-4B and Qwen3.6-35B-A3B on the same org

Technical Details

• Original Model: Qwen/Qwen3-Embedding-4B
• Architecture: Qwen3 Causal LM → last-token pooling
• Parameters: 4.02 B
• Embedding dim: up to 2560 (Matryoshka)
• Context: 32 K
• License: Apache 2.0
• Quantized with: llama.cpp build bafae2765
• Quantized by: BatiAI

BatiAI RAG Stack

License

Mirrors upstream Qwen Apache 2.0 — commercial use permitted.