bge-reranker-v2-m3-fp16emb — lossless FP16 embedding tables (1.74 GB), tuned for oMLX on Apple Silicon

English · 中文

BAAI/bge-reranker-v2-m3 (multilingual cross-encoder: XLM-RoBERTa-large, 568M parameters, one relevance logit per query–document pair) with exactly one change: the three embedding tables (word / position / token-type) are stored in FP16, everything else stays FP32. The upstream checkpoint is FP32 on disk, but all 567,755,777 of its parameters are exactly representable in FP16 (we checked element by element), so nothing is lost: 2.27 GB → 1.74 GB. transformers loads it as FP32 and reproduces the original scores; on oMLX 0.7.0 a 24-document request costs the same (+1.6 %, inside FP32's own run-to-run spread) with identical ranking.

We deliberately stopped there. Converting the linear layers too (all-FP16) is slower on oMLX 0.7.0 — +9.9 % latency when a large model is resident — and quantized (8/4-bit) weights do not load in oMLX 0.7.0's reranker path. The rest of this card is what we measured while getting there, plus tested guidance for using the reranker well (long documents, request size, candidate count, thresholds) and a client script.

At a glance

Lineage: BAAI/bge-reranker-v2-m3 revision 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e (2024-06-24; model.safetensors SHA-256 d9e3e081faff1eefb84019509b2f5558fd74c1a05a2c7db22f74174fcedb5286 matches the official LFS hash; tokenizer files match) → recipe/make_fp16_embeddings.py → this repository (model.safetensors SHA-256 e82c87f7783b42527d9e335c8a667cd606fe1c53eff51384a13d1f95a2c85ef4). config.json is the upstream file, unchanged — its torch_dtype still says float32; the mixed precision lives in the tensor dtypes only.

Measured results

Mac Studio M1 Ultra (64-core GPU) 64 GB, macOS 15.8, oMLX 0.7.0 (MLX 0.32.2), one machine. Performance numbers go through the server's own RerankerEngine / real HTTP, ABBA-interleaved, and — the part that changed our decision — with a large LLM resident, which is the everyday state.

Which precision?

Why all-FP16 loses: oMLX's native XLM-R builds the attention mask in FP32, so FP16 linear weights are up-cast on every request, and the engine clears the MLX buffer pool after each request — measured alone the penalty is only +1.5 %, but with a large model resident the ≈1.2 GB of temporary buffers hurt (+9.9 %). The embedding tables are only gathered by row, so keeping them in FP16 is free. (We first shipped all-FP16 on the stand-alone numbers and reverted it after the resident test — hence the extra column.) Upstream PR #4168 (in oMLX v0.7.1.dev1; stable is still 0.7.0) changes this: see the preview below.

Quality: reranking vs the first stage

48 queries (zh→zh, en→en, zh↔en) over 315 technical-document chunks; per query ≈ 40 candidates = the gold chunk + the 30 hardest non-gold chunks of a two-retriever first stage + 8 random ones (1874 pairs).

Reranking adds ΔMRR@10 +0.100 over the fused first stage (95 % CI [+0.009, +0.194], 18 wins / 12 losses), more for cross-lingual queries (MRR 0.496 → 0.675). Numerically, oMLX FP32 differs from the PyTorch reference by ≤ 5.2e-6, this repo by ≤ 2.3e-4 (FP16 summation of the embedding rows inside MLX), all-FP16 by ≤ 4.0e-4 — no pair reversed by more than 0.01, no threshold flip. The test set is small (48 queries), so differences between variants are judged mostly by fidelity, not by ranking metrics.

Long documents: chunk them (oMLX 0.7.0 truncates at 512 tokens)

"Needle in a haystack": one positive document of ≈ 1.4K tokens (5 distractor blocks + the gold block at a random position) against 15 hard-negative documents; 96 trials.

Every chunking variant beats truncation (ΔMRR +0.17 … +0.25; intervals exclude 0); window size (128–448) and aggregation (max / top-2 mean / log-sum-exp) are statistically indistinguishable; overlap only costs time. Under truncation an answer in the 3rd block or later is almost never found (hit rate by block: 0.57 / 0.45 / 0.14 / 0.06 / 0.06 / 0.00). Use client/chunked_rerank.py — in a smoke test with the answer ≈ 1000 tokens into a 1.7K-token document, plain /v1/rerank scored all four documents ≈ 0.0001, the chunked client scored the right one 0.968.

Request size, chat-stream pauses, candidate count (oMLX 0.7.0)

• Latency is linear in the number of pairs (≈ 27 ms per pair at 512 tokens: 8 / 24 / 32 / 64 / 128 docs → 251 / 656 / 861 / 1734 / 3418 ms); beyond ≈ 24 documents each document gets only 3 % cheaper. Keep a request to ≤ 32 documents.
• The reranker and LLM decoding share one MLX thread: a rerank request pauses a streaming chat for about its own duration (longest pause for 8 / 24 / 32 / 64 / 128 docs: 0.18 / 0.64 / 0.88 / 1.5 / 3.0 s). With a chat stream running keep requests ≤ 16 documents; 128 documents sent as 16 × 8 cost +19 % total time but the longest pause drops to 0.5 s.
• Requests are fully serialized (1 / 2 / 4 / 8 concurrent requests: wall 0.43 / 0.84 / 1.65 / 3.25 s) — parallel sends only queue. Bucketing documents by length is 2 % slower.
• oMLX 0.7.0 has no cap on the number of documents: 1000 documents of 512 tokens need ≈ +45 GB. Cap it in your client.
• How many candidates? On our set the first stage's recall@K is 0.896 / 0.938 / 0.958 / 0.958 / 1.000 for K = 6 / 8 / 10 / 12 / 16, and the reranked MRR@10 is flat from K = 10 to 30 (0.692 / 0.694 at K = 12 / 16, 0.691 at 24). K = 16 gave the same quality as 24 in about ⅔ of the time (≈ 275 vs 412 ms). Check recall on your own corpus before copying that.
• Absolute score thresholds lose answers: the median score of the gold chunk is 0.817, but 8 of 48 queries (17 %) have a gold score below 0.2 (and 2 of 48 below 0.05) — mostly queries whose wording is far from the text. A "keep only score ≥ 0.2" gate would drop them.

Preview of the upstream fix (PR #4168)

oMLX v0.7.1.dev1 (a development release; the stable release is still 0.7.0) contains PR #4168: /v1/rerank forwards max_length (default = tokenizer limit 8192), length-sorted batches, and XLM-R attention through mx.fast.scaled_dot_product_attention with the mask cast to the activation dtype. We applied the PR to an isolated copy of the oMLX package (production app untouched) and re-ran the same tests: all-FP16 becomes the fastest build — 24 docs p50 414 → 306 ms (−26 %), footprint 4.02 → 2.39 GB (−40 %), long documents scored whole (R@1 0.417 vs 0.229 truncated; not significantly different from chunking). One of 1874 pairs crossed the 0.05 threshold with all-FP16, so we will re-run the gates on the first stable release and then switch; until then this repository (FP32 linear layers) is the right build for oMLX 0.7.0. Plain transformers is not affected by any of this.

Recommended oMLX settings

omlx/model_settings.entry.json (key = directory name):

{ "ttl_seconds": 3600, "model_alias": "BGE-Reranker-V2-M3" }

There is nothing else to tune on the server side in 0.7.0: /v1/rerank does not forward max_length, a request is one forward pass, requests are serialized. Not pinned: the first request after a cold load takes ≈ 1.9 s, and oMLX's engine pool may evict an idle reranker anyway when a large model needs the room.

Quick start

HF_ENDPOINT=https://hf-mirror.com hf download YCF-AI/bge-reranker-v2-m3-fp16emb --local-dir ~/models/bge-reranker-v2-m3
# oMLX: put the folder under a model_dirs entry, merge omlx/model_settings.entry.json (key = folder name), reload. Then:
curl http://127.0.0.1:8000/v1/rerank -H "Authorization: Bearer $OMLX_API_KEY" -H "Content-Type: application/json" \
     -d '{"model": "BGE-Reranker-V2-M3", "query": "what is a panda?", "documents": ["The giant panda is a bear native to China.", "Paris is the capital of France."]}'

Long documents / many candidates: python client/chunked_rerank.py --tokenizer ~/models/bge-reranker-v2-m3/tokenizer.json --query "..." a.txt b.txt, or from chunked_rerank import rerank (192-token windows, ≤ 32 per request, max score per document).

Plain transformers (tested; the FP16 tables are cast to FP32 on load, scores equal the original):

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "YCF-AI/bge-reranker-v2-m3-fp16emb"
tok, model = AutoTokenizer.from_pretrained(repo), AutoModelForSequenceClassification.from_pretrained(repo).eval()
pairs = [["what is a panda?", "The giant panda is a bear native to China."], ["what is a panda?", "Paris is the capital of France."]]
with torch.no_grad():
    batch = tok(pairs, padding=True, truncation=True, max_length=512, return_tensors="pt")
    scores = torch.sigmoid(model(**batch).logits.view(-1))      # relevance probability in [0, 1]

Rebuild recipe

HF_ENDPOINT=https://hf-mirror.com hf download BAAI/bge-reranker-v2-m3 --revision 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e --local-dir up
shasum -a 256 up/model.safetensors      # d9e3e081faff1eefb84019509b2f5558fd74c1a05a2c7db22f74174fcedb5286
python recipe/make_fp16_embeddings.py up out      # needs mlx; aborts unless every table survives FP32 → FP16 → FP32 unchanged
shasum -a 256 out/model.safetensors     # e82c87f7783b42527d9e335c8a667cd606fe1c53eff51384a13d1f95a2c85ef4

Verified 2026-10-10 with MLX 0.32.2: byte-identical model.safetensors; config.json and tokenizer files are copied unchanged.

Known limits

• oMLX 0.7.0 truncates each pair at 512 tokens and has no candidate cap (see above); this is the server, not the model.
• Scores are specific to this model: do not carry thresholds over from other rerankers. We did not evaluate other rerankers (e.g. Qwen3-Reranker, jina-reranker).
• One machine, 48 queries over technical documents (zh / en); candidate-count and threshold results need re-checking on your corpus. Cold-disk load times were not measured.
• The best build depends on the host runtime: after an oMLX release that contains #4168 an all-FP16 build will be better, and this repository will be updated then.

License and attribution

Apache-2.0, inherited from BAAI/bge-reranker-v2-m3 (original card: README-upstream.md, with its assets/). Please cite the original work: BGE M3-Embedding (Chen et al., 2024, arXiv:2402.03216) and Making Large Language Models A Better Foundation For Dense Retrieval (Li et al., 2023, arXiv:2312.15503). Part of a set of notes on running a local model stack on an M1 Ultra: tuning notes and data · embedding companion: bge-m3-8bit-MLX.

---

中文说明

English · 中文

BAAI/bge-reranker-v2-m3（多语言交叉编码器：XLM-RoBERTa-large，5.68 亿参数，每个查询—文档对输出一个相关性 logit）只做了一处改动：三张嵌入表（词 / 位置 / 类型）存为 FP16，其余张量保持 FP32。 上游检查点在磁盘上是 FP32，但它全部 567,755,777 个参数都恰好能用 FP16 精确表示（逐元素验证），所以转换无损：2.27 GB → 1.74 GB。transformers 加载时按 FP32 读入，分数与原件一致；在 oMLX 0.7.0 上 24 篇候选的请求耗时不变（+1.6 %，在 FP32 自身波动范围内），排序完全相同。

我们有意只做到这一步：把线性层也转 FP16（全 FP16）在 oMLX 0.7.0 上更慢（大模型常驻时 延迟 +9.9 %），而量化（8/4-bit）权重在 oMLX 0.7.0 的重排加载路径上根本加载不了。本卡其余部分是过程中的实测，以及经过实测的使用建议（长文、请求大小、候选数、阈值）和一个客户端脚本。

要点

关键实测结论

• 选型：24 篇 × 48 请求，真实服务 HTTP，ABBA 交错，CyberTiel（28.5 GB）常驻：FP32 446.5 ms，本方案 453.5 ms（+1.6 %），全 FP16 490.5 ms（+9.9 %）。全 FP16 单独运行只慢 1.5 %，大模型常驻时慢 9.9 %（约 1.2 GB 的升精度临时缓冲每个请求后被清掉再重分配）。我们最初仅凭单独运行的数据上线过全 FP16，同驻测试暴露问题后撤回——所以表里有「大模型常驻」这一列。嵌入表按行取用，存 FP16 没有代价。
• 质量：48 查询 / 1874 对，PyTorch FP32 参考 R@1 0.542、MRR@10 0.689；本方案与 FP32、全 FP16 在 oMLX 上排序指标逐位相同。重排相对一阶段融合 ΔMRR@10 +0.100（95 % CI [+0.009, +0.194]），跨语言查询提升更大（MRR 0.496 → 0.675）。
• 长文必须分块：96 组「大海捞针」中，截断 512 的 R@1 只有 0.229；按 192 token 窗口、不重叠、取最大分 为 0.500。所有分块配置都显著优于截断，彼此无显著差异，重叠 25 % 无益且多花约 25 % 时间。用 client/chunked_rerank.py：冒烟测试里答案位于 1.7K token 文档约 1000 token 处，直接调用四篇都 ≈ 0.0001，分块客户端给正确文档 0.968。
• 请求形状：延迟随对数线性（≈ 27 ms/对）——单请求 ≤ 32 篇；重排与 LLM 解码共用一个 MLX 线程，请求多长聊天流就停多久（8/24/32/64/128 篇 → 最长停顿 0.18/0.64/0.88/1.5/3.0 s），有聊天流时 ≤ 16 篇，128 篇拆成 16×8 总耗时 +19 % 但最长停顿降到 0.5 s；请求被完全串行化，并行发送只会排队；按长度分桶反而慢 2 %；0.7.0 没有候选数上限保护（1000 篇 ×512 token 约 +45 GB），请在客户端限制。
• 一阶段候选数：召回@K 在 K=16 达 100 %，重排后 MRR 与 K=24 相同、耗时约少 ⅓（≈ 275 vs 412 ms）——这是我们这套 48 条技术文档查询上的结果，换语料先看召回。
• 绝对阈值会丢答案：金标分数中位数 0.817，但 8/48（17 %）的查询金标分 < 0.2、2/48 < 0.05（多是措辞与原文差距大的查询）；「分数 ≥ 0.2 才保留」会把它们丢掉。
• 上游修复预演（PR #4168，含于 oMLX v0.7.1.dev1，稳定版仍是 0.7.0）：在隔离副本里打补丁（生产应用未动）：全 FP16 成为最快方案——24 篇 414 → 306 ms（−26 %），footprint 4.02 → 2.39 GB（−40 %），长文可整篇评分（R@1 0.417，与分块无显著差异）。全 FP16 在 1874 对里有 1 对越过 0.05 阈值，所以等首个含该修复的稳定版发布后重做门禁再切换，届时更新本仓库；在此之前，本仓库（线性层 FP32）是 oMLX 0.7.0 的正确选择。纯 transformers 不受影响。

推荐 oMLX 设置

见 omlx/model_settings.entry.json（键名=目录名）：ttl_seconds 3600、别名 BGE-Reranker-V2-M3。0.7.0 里服务端没有别的可调项（/v1/rerank 不转发 max_length；一个请求一次前向；请求之间串行）。不 pinned：冷加载后首请求约 1.9 s，引擎池本来也会在大模型需要空间时逐出空闲的重排器。

快速开始与复现

下载、oMLX 调用、transformers 代码（已实测）与分块客户端用法见上方英文部分。复现：固定上游 revision 下载 → 校验 model.safetensors SHA-256 → python recipe/make_fp16_embeddings.py up out（只有每张表 FP32→FP16→FP32 完全不变时才会转换，否则中止）；2026-10-10 用 MLX 0.32.2 验证与本仓库 model.safetensors（e82c87f7…）逐字节一致，config.json 与分词器文件原样复制。

已知限制

• oMLX 0.7.0 对每对截断 512 token、无候选数上限——这是服务端的限制，不是模型的。
• 分数刻度只属于这个模型：不要把其他重排器的阈值搬过来。我们没有评测其他重排模型（如 Qwen3-Reranker、jina-reranker）。
• 单机、48 条中英技术文档查询；候选数与阈值结论请在你的语料上复核；未测冷盘（页缓存冷）的加载时间。
• 最优构建取决于宿主运行时：含 #4168 的 oMLX 发布后，全 FP16 会更优，届时本仓库会更新。

许可与致谢

Apache-2.0，继承自 BAAI/bge-reranker-v2-m3（原卡见 README-upstream.md，配图在 assets/）。请引用原论文：BGE M3-Embedding（Chen et al., 2024，arXiv:2402.03216）与 Making Large Language Models A Better Foundation For Dense Retrieval（Li et al., 2023，arXiv:2312.15503）。 本模型属于一套 M1 Ultra 本地模型栈调优笔记：笔记与数据 · 配套嵌入模型：bge-m3-8bit-MLX。