multilingual-reranker-small v1.2 — GGUF (Q8_0)

A GGUF of Horizon-Labs/multilingual-reranker-small (v1.2, commit 23139fec6f1f448cdf89a5ac2d2c6bf5e8c7b398), for llama.cpp: a multilingual cross-encoder (141M, mmBERT-small) that scores how well a passage answers a question. Q-P1NG and Q-P1NG Node use it to order the passages their archive finds.

Use

llama.cpp with pooling rank (for example llama-server -m multilingual-reranker-small-v1.2-Q8_0.gguf --reranking). The pair is [BOS] question [EOS] passage [EOS]; the score is a logit (higher: more relevant; sigmoid gives 0-1).

Measured

On 35 Italian questions about two PDFs (194 pages), the 10 passages found by a hybrid search (BM25 and Granite Embedding 107M) reordered by it: Hit@1 from 25 to 31 on a laptop CPU (about 1 s for ten passages), from 28 to 33 on a Pixel phone (within 2 s).

Conversion

1. convert_hf_to_gguf.py of llama.cpp (the version Q-P1NG builds), weights quantized to Q8_0.
2. The metadata key tokenizer.ggml.metaspace.prepend_scheme set to first instead of always: llama.cpp refuses always for BPE vocabularies, and for a question and a passage each tokenized alone the two give the same tokens.

Checked against transformers (the model's own RoPE bases): identical tokens; scores within 0.002 in F16 and 0.06 in Q8_0.

Licence

Apache-2.0, as the original model: see multilingual-reranker-small-LICENSE.txt and multilingual-reranker-small-NOTICE.txt (mmBERT-small, MIT). The original was trained on openly licensed web text (FineWeb, FineWeb-2: ODC-BY), MIRACL (Apache-2.0) and WebFAQ (CC-BY-4.0), with labels from Qwen3-Reranker-4B (Apache-2.0). The changes made here are listed in multilingual-reranker-small-CONVERSION.txt.