Nemotron-3.5-Embed-8B-GGUF

GGUF conversion and quantizations of NVIDIA's Nemotron-3.5-Embed-8B-BF16, a multilingual multimodal (text + image) embedding model. Converted and quantized with llama.cpp.

Includes the BF16 Pixtral vision projector, so document-page images and image + text inputs work alongside text. The text tower is the frozen Nemotron-3-Embed-8B language model, so it shares the embedding space of the text-only model.

Files

File	Quantization	Size	Notes
`Nemotron-3.5-Embed-8B-BF16.gguf`	BF16	15.91 GB	reference, same weights as the base model
`Nemotron-3.5-Embed-8B-Q8_0.gguf`	Q8_0	8.46 GB	near-lossless
`Nemotron-3.5-Embed-8B-Q6_K.gguf`	Q6_K	6.53 GB	good quality/size trade-off
`Nemotron-3.5-Embed-8B-Q4_K_M.gguf`	Q4_K_M	4.90 GB	smallest, most quality loss
`mmproj-Nemotron-3.5-Embed-8B-BF16.gguf`	BF16 projector	0.87 GB	Pixtral vision projector (required for images)

Use the model's task prefixes: query: for queries, passage: for documents.

llama.cpp — text embeddings

llama-embedding -m Nemotron-3.5-Embed-8B-Q8_0.gguf \
  -p "query: What is NVIDIA NeMo Framework?" \
  --pooling mean --ctx-size 512

Output is a single 4096-dimensional, L2-normalized vector.

The GGUF header carries the model's full 262,144-token context, which makes llama.cpp allocate enormous compute buffers. Always pass a realistic --ctx-size (512 for short queries, up to 32768 for long documents) — otherwise allocation can fail.

llama.cpp — image + text embeddings

llama-server \
  -m Nemotron-3.5-Embed-8B-Q8_0.gguf \
  -mm mmproj-Nemotron-3.5-Embed-8B-BF16.gguf \
  --embedding --pooling mean --ctx-size 8192 -ngl 0 --port 8099

curl http://127.0.0.1:8099/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": [{
      "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
        {"type": "text", "text": "query: What is shown in this screenshot?"}
      ]
    }]
  }'

Returns one 4096-dim, L2-normalized vector. Image + text produces a different vector than text alone.

Ollama

Published as maternion/nemotron-3.5-embed, tags :8b (Q4_K_M, the default alias), :8b-q4_k_m, :8b-q8_0, :8b-q6_k. Every tag bundles the BF16 projector, so ollama show reports vision and /api/embed accepts map-shaped multimodal input:

IMG=$(base64 -w0 screenshot.png)
curl http://localhost:11434/api/embed -d "{
  \"model\": \"maternion/nemotron-3.5-embed:8b\",
  \"input\": {\"text\": \"query: What is shown in this screenshot?\", \"image\": \"$IMG\"}
}"

BF16 is HF-only; it is not published as an Ollama tag.

Model details

Parameters: 8.38B total
Text architecture: Ministral-3 (34 layers, hidden 4096, 32 attention heads / 8 KV heads, head_dim 128, FFN 14336, vocab 131072)
Embedding: 4096-dim, mean-pooled, L2-normalized
Dynamic embedding size: truncate from the end (e.g. first 1024 or 512 dims) and re-normalize; truncated vectors remain effective for retrieval
Vision projector: Pixtral, 24 blocks, hidden 1024, image size 1540 px (default preprocessing longest edge 1120 px), projection dim 4096
Context: 32,768 tokens
Languages: 34 for text, 18 for images

Benchmarks

NDCG@10:

Model	ViDoRe V3 image	ViDoRe V3 text	ViDoRe V3 image+text	MIRACL-VISION image
Qwen3-VL-Embedding-2B	52.89	48.31	51.97	58.42
Nemotron-3.5-Embed-1B-BF16	58.02	57.80	58.82	68.93
Qwen3-VL-Embedding-8B	58.29	53.69	57.03	68.33
Nemotron-3.5-Embed-8B-BF16	60.39	60.65	61.47	72.64

Published BF16 scores from the base model; quantized variants may differ slightly.

License

OpenMDW-1.1. The Ministral-3-8B-Instruct-2512-BF16 base model is Apache 2.0.

maternion/Nemotron-3.5-Embed-8B-GGUF