bge-reranker-v2-m3, 8-bit MLX
BAAI's multilingual reranker BAAI/bge-reranker-v2-m3 (an XLM-RoBERTa-large cross-encoder, 568M parameters) with its weights quantized to 8 bits in MLX's format, for Apple Silicon. It reads a query and a passage together and returns one relevance logit; a sigmoid turns it into a 0 to 1 score. The download drops from 2.3 GB of float32 to 0.63 GB, and the scores stay within 0.07 logits of the original on the pairs measured below.
Files
The model sits at the repository root, in the layout MLX loaders expect:
config.json BAAI's config plus "quantization": {"group_size": 64, "bits": 8}
model.safetensors 611 MB, MLX tensors (685 tensors, 146 of them 8-bit)
tokenizer.json BAAI's tokenizer, unchanged
tokenizer_config.json BAAI's, unchanged
special_tokens_map.json BAAI's, unchanged
rerank.py a self-contained MLX implementation and command-line example
conversion/convert_reranker.py the conversion script
LICENSE Apache License 2.0
What the quantization does
mx.quantize with 8 bits and a group size of 64 (affine quantization: each
group of 64 weights stores a float16 scale and bias) is applied to:
- the word embeddings (250,002 x 1024),
- every Linear layer of the 24 encoder layers (query, key, value, attention output, intermediate, output),
- the classifier's dense layer.
Everything else is converted to float16 and not quantized: position and
token-type embeddings, all LayerNorms, all biases, and the classifier's final
1024-to-1 projection, which produces the logit. The tensor names are BAAI's, so
a loader builds the model as usual and swaps in QuantizedLinear and
QuantizedEmbedding wherever a .scales tensor exists.
Quality check against the original
Twenty query/passage pairs (five queries in English, Russian and German with
four candidate passages each: one answer, one near miss, two unrelated)
scored three ways: BAAI's float32 weights in PyTorch (transformers 5.22,
torch 2.15), the same float32 weights in rerank.py on MLX, and this 8-bit
checkpoint in rerank.py.
| Max abs logit difference vs PyTorch | Mean | Max sigmoid difference | Same top passage | Same full order | |
|---|---|---|---|---|---|
| MLX, BAAI float32 | < 0.001 | < 0.001 | < 0.0001 | 5/5 | 5/5 |
| MLX, this 8-bit | 0.067 | 0.028 | 0.0095 | 5/5 | 5/5 |
The logits run from -11 to +7.6. Batching with padding changes the 8-bit scores by at most 0.010.
The checkpoint was also compared in a private retrieval evaluation: 48 questions over two personal Markdown note collections (the larger has 163 notes), candidates from hybrid BM25 and embedding search, 512 tokens per pair, scored by mlx-swift-lm's MLXRerankers:
| Variant | model.safetensors | MRR@6, set 1 | MRR@6, set 2 | Expected note's rank vs float32 |
|---|---|---|---|---|
| no reranker | 0.804 | 0.829 | ||
| float32 (BAAI) | 2.27 GB | 0.896 | 0.938 | |
| float16 | 1.14 GB | 0.896 | 0.938 | same on 48/48; all top-6 lists identical |
| 8-bit (this repository) | 0.61 GB | 0.896 | 0.938 | same on 48/48; top-6 lists differ on 10 questions, the top source on 1 |
Speed
rerank.py on an Apple M2 Pro (32 GB), MLX 0.32.4: one query against 16
passages of 111 tokens per pair takes 231 ms (BAAI's float32 weights: 251 ms).
The 8-bit weights mostly save memory and download size.
Usage
Python (MLX)
pip install mlx tokenizers huggingface_hub
python rerank.py "what is panda?" "hi" \
"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China."
From code:
from huggingface_hub import snapshot_download
from rerank import load, score
model, tokenizer = load(snapshot_download("nikioth/bge-reranker-v2-m3-mlx-8bit"))
passages = ["hi", "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China."]
logits = score(model, tokenizer, "what is panda?", passages)
# about [-8.2, 5.3]; 1 / (1 + exp(-logit)) gives a 0..1 relevance
rerank.py encodes each pair as <s> query </s></s> passage </s> with the
tokenizer's template, truncates pairs to 512 tokens and pads batches. The same
loader also reads BAAI's original float32 checkpoint.
Swift (mlx-swift-lm)
The checkpoint loads with the MLXRerankers library of
mlx-swift-lm (3.32), from a local
copy of this repository:
import MLXHuggingFace
import MLXRerankers
let container = try await RerankerModelFactory.shared.loadContainer(
from: directory, using: #huggingFaceTokenizerLoader())
let response = try await container.scores(
query: "what is panda?", documents: passages, instruction: nil,
options: RerankExecutionOptions(maxBatchSize: passages.count, maxBatchTokens: 512, truncation: .truncate))
for result in response.results { print(result.index, result.score) }
Limitations
- The checks above are small: 20 pairs in three languages, and one private retrieval set. Verify on your own data before relying on it for ranking decisions where small score differences matter.
- Inputs longer than 512 tokens per pair were not tested, although the model accepts up to 8192 positions.
- MLX only: this is not a PyTorch or GGUF checkpoint.
Reproducing the conversion
pip install mlx huggingface_hub
python conversion/convert_reranker.py --out bge-reranker-v2-m3-mlx-8bit --bits 8
The script downloads BAAI/bge-reranker-v2-m3 at revision
953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e, checks the SHA-256 of its weights
and tokenizer, and writes the five model files. With MLX 0.32.4 the output is
byte for byte identical to this repository's files. Without --bits it writes
a float16 copy instead.
License and attribution
The model is BAAI's bge-reranker-v2-m3, released under the
Apache License 2.0 (LICENSE).
Changes made by nikioth: the weights were converted from float32 to MLX
safetensors, with the layers listed above quantized to 8 bits and the rest
stored as float16, and config.json gained a quantization entry and a
torch_dtype of float16. The tokenizer files are BAAI's, unchanged. The
scripts here are also Apache-2.0. Provided as is, without warranties of any
kind.
Citation
@misc{li2023making,
title = {Making Large Language Models A Better Foundation For Dense Retrieval},
author = {Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
year = {2023},
eprint = {2312.15503},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
@misc{chen2024bge,
title = {BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text
Embeddings Through Self-Knowledge Distillation},
author = {Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
year = {2024},
eprint = {2402.03216},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}