NVIDIA Nemotron-3.5-Rerank-8B-BF16 converted to GGUF for llama.cpp, in BF16, Q8_0 and Q4_K_M. A reranker: it scores a query against a passage and returns a relevance logit. This build is text only.
Built by NetProjects. Not affiliated with NVIDIA. All credit for the model goes to NVIDIA; this repository only changes the format and the precision.
Linear(4096 -> 1) score head. The Pixtral vision tower is removed.query: {query}\n\npassage: {document}cls.output and a built-in rerank template. It gives native /v1/rerank and needs llama.cpp with native-rerank.patch.score_weight.npy; score = dot(score_weight, embedding). It works with stock llama.cpp and rerank_wrapper.py.| File | Quant | Size | Runs on |
|---|---|---|---|
| Nemotron-3.5-Rerank-8B-rank-BF16.gguf | BF16 | 15.9 GB | patched llama.cpp |
| Nemotron-3.5-Rerank-8B-rank-Q8_0.gguf | Q8_0 | 8.5 GB | patched llama.cpp |
| Nemotron-3.5-Rerank-8B-rank-Q4_K_M.gguf | Q4_K_M | 4.9 GB | patched llama.cpp |
| Nemotron-3.5-Rerank-8B-embed-BF16.gguf | BF16 | 15.9 GB | stock llama.cpp + wrapper |
| Nemotron-3.5-Rerank-8B-embed-Q8_0.gguf | Q8_0 | 8.5 GB | stock llama.cpp + wrapper |
| Nemotron-3.5-Rerank-8B-embed-Q4_K_M.gguf | Q4_K_M | 4.9 GB | stock llama.cpp + wrapper |
Q8_0 is the recommended choice.
/v1/rerank (rank files)Upstream llama.cpp does not register Mistral3ForSequenceClassification yet, so apply native-rerank.patch (made against llama.cpp master 0ec4027):
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 0ec4027
git apply ../native-rerank.patch
cmake -B build -DGGML_VULKAN=ON # or CUDA / Metal / CPU
cmake --build build --config Release -j
Local server (Cohere/Jina-style /v1/rerank):
llama-server -m Nemotron-3.5-Rerank-8B-rank-Q8_0.gguf --reranking -ngl 99 -c 4096 -ub 4096 -b 4096 --port 8095
curl http://127.0.0.1:8095/v1/rerank -H "Content-Type: application/json" -d '{
"query": "Where can I see the Parthenon sculptures?",
"documents": ["The Acropolis Museum in Athens displays the Parthenon sculptures.", "A heat pump moves heat."]
}'
relevance_score is the raw logit.
The patch registers Mistral3/Ministral3ForSequenceClassification in the converter, maps score.weight to cls.output, and writes pooling RANK, labels ["score"] and the rerank template. It also adds CLS_OUT to the MISTRAL3 tensors (as an optional cls_out tensor) and defaults the pooling cls to MEAN in mistral3.cpp.
llama-server -m Nemotron-3.5-Rerank-8B-embed-Q8_0.gguf --embedding --pooling mean -ngl 99 --port 8095
Send the formatted prompt with add_special: false; score = dot(score_weight.npy, embedding). rerank_wrapper.py is a minimal example and expects score_weight.npy (and a pairs.json test file) next to it.
All numbers below were measured on these files. The reference is the original model run layer by layer in bf16 PyTorch on CPU.
20 query/passage pairs (10 relevant, 10 irrelevant):
| Quant | Max score difference | Mean difference | Pearson | Spearman | Pairwise order | Relevant > irrelevant |
|---|---|---|---|---|---|---|
| BF16 | 0.0113 | 0.0034 | 0.99999 | 1.0000 | 1.0000 | 10/10 |
| Q8_0 | 0.0164 | 0.0083 | 0.99999 | 1.0000 | 1.0000 | 10/10 |
| Q4_K_M | 0.0356 | 0.0130 | 0.99990 | 0.9985 | 0.9947 | 10/10 |
Native /v1/rerank scores from the rank files match the wrapper scores from the embed files exactly (max difference 0.0000). The raw numbers are in parity.csv and parity_native.csv.
[ and <. Native /v1/rerank does not do this; rerank_wrapper.py does. This only matters for text containing those characters.The model is NVIDIA's, released under the OpenMDW License Agreement, version 1.1; the LICENSE file in this repository is that agreement, kept as it requires. Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. It is built on mistralai/Ministral-3-8B-Instruct-2512-BF16 (Apache-2.0); see NOTICE and THIRD_PARTY_NOTICES.md.
Modification notice: these files were converted to GGUF and quantized with llama.cpp. The vision tower was removed, and in the embed files the score head was split out into score_weight.npy. They are not official NVIDIA releases and are provided AS IS, without warranty.
Built and tested on ASUS ProArt (AMD Radeon 8060S, Vulkan) · llama.cpp 0ec4027
Also from NetProjects: MLX builds of this model: Nemotron-3.5-Rerank-8B-mlx-8bit · Nemotron-3.5-Rerank-8B-mlx-4bit · NetProjects MLX builds collection · all models at huggingface.co/netprojects
nvidia/Nemotron-3.5-Rerank-8B-BF16