Qwen3-Embedding-8B GGUF — Quantized by BatiAI

GGUF quantizations of Qwen/Qwen3-Embedding-8B — MTEB #1 among open embeddings (70.58 avg) at release. Top-tier retrieval quality for users with 24 GB+ Macs. Part of BatiAI's on-device RAG stack for BatiFlow.

Why 8B?

• MTEB Multilingual avg 70.58 — #1 among open-source embedding models at release
• Strong Korean + multilingual retrieval — single model covers 100+ languages
• Matryoshka — 2560 → 1024 → 512 → 256 dimensions at runtime, no re-embed

Quick Start

Ollama

ollama pull batiai/qwen3-embedding:8b        # Q6_K (6.2 GB, recommended default)
ollama pull batiai/qwen3-embedding:8b-q8     # Q8_0 (8.0 GB, max quality)

curl http://localhost:11434/api/embeddings -d '{
  "model": "batiai/qwen3-embedding:8b",
  "prompt": "semantic search query"
}'

llama.cpp

./llama-server \
  -m Qwen3-Embedding-8B-Q8_0.gguf \
  --embeddings --pooling last -c 32768 \
  --host 127.0.0.1 --port 8080

Available Quantizations

Why Q6 over Q8 as default? Drift 0.999 means the two files produce functionally identical embeddings. On a 24 GB Mac, saving ~1.8 GB keeps the window manager and browser comfortable alongside inference. If you want maximum conservatism, pull :8b-q8.

Why no IQ3 / IQ4 for embedding? Embedding quality cascades into cosine-similarity drift at low bit-widths. Q6_K / Q8_0 are the safe range.

Quality Verification (measured)

Four-stage harness run on both quants. Full testset + script reproducible via scripts/bench-embedding-quality.sh.

Perfect retrieval across 40 queries on both languages — the full value of MTEB #1 shows up cleanly on this harness. Q8_0 and Q6_K produce essentially identical quality; pick Q6_K if disk is tight.

Quality tier comparison (across BatiAI text-embedding lineup)

Cleanly monotonic: every metric improves with size. If retrieval accuracy is critical and you have the RAM, 8B is the pick.

Matryoshka — runtime-configurable dimension

Qwen3-Embedding-8B outputs up to 2560 dimensions. BatiFlow RAG stack defaults to 1024 (quality / latency sweet spot). Truncate at read time:

emb = get_embedding(text)     # [2560]
emb_1024 = emb[:1024]
import numpy as np
emb_1024 = emb_1024 / np.linalg.norm(emb_1024)

Why text-only?

For image + text retrieval, see Qwen3-VL-Embedding-8B on BatiAI.

RAG Stack Integration

user query
   ↓ [Qwen3-Embedding 8B]            ← YOU ARE HERE (top tier, MTEB #1)
1024-dim vector
   ↓ vector DB (sqlite-vec / LanceDB)
top-K candidates
   ↓ [Qwen3-Reranker 8B]
top-3
   ↓ [Qwen3.6-35B-A3B chat LLM]
answer

Recommended Usage — query vs document

# Query side
query = "Instruct: Given a document query, retrieve the most relevant chunk.\n" \
        "Query: " + user_input

# Document side — no instruction prefix
document = chunk_text

BatiFlow handles this automatically. For custom integrations, see Qwen3-Embedding usage.

Why BatiAI?

• Quantized directly from Qwen's BF16 safetensors — no intermediate GGUF
• general.author: BatiAI for provenance
• 4-stage quality harness — full EN + KO retrieval numbers published
• Matched with Qwen3-Reranker-8B and Qwen3.6-35B-A3B — complete Mac-first RAG stack

Technical Details

• Original Model: Qwen/Qwen3-Embedding-8B
• Architecture: Qwen3 Causal LM → last-token pooling
• Parameters: 8.19 B
• Embedding dim: up to 2560 (Matryoshka)
• Context: 32 K
• License: Apache 2.0
• Quantized with: llama.cpp build bafae2765
• Quantized by: BatiAI

BatiAI RAG Stack

License

Mirrors upstream Qwen Apache 2.0 — commercial use permitted.