KaLM-Reranker-V1-Small Q4_K_M GGUF

[!IMPORTANT] This model requires the patched llama.cpp runtime bundled in llama.cpp/. Stock llama.cpp does not recognize the t5gemma2 architecture used here. Ollama, LM Studio, llama-server and other stock frontends are not supported by this release.

This is the text-only Q4_K_M GGUF conversion of KaLM-Embedding/KaLM-Reranker-V1-Small.

Model file

Architecture: t5gemma2; physical tensors: 679; text parameters: 1,697,782,016. The tokenizer is embedded in the GGUF.

Download

The custom CLI requires a local model path and has no -hf option:

hf download KaLM-Embedding/KaLM-Reranker-V1-Small-Q4_K_M-GGUF \
  --local-dir KaLM-Reranker-V1-Small-Q4_K_M-GGUF
cd KaLM-Reranker-V1-Small-Q4_K_M-GGUF
sha256sum --check SHA256SUMS

Build the required runtime

git clone https://github.com/ggml-org/llama.cpp llama.cpp-src
git -C llama.cpp-src checkout 277a105dc8f8643dab54331926a9830860a03292
bash "$PWD/llama.cpp/apply-patches.sh" "$PWD/llama.cpp-src"
cmake -S llama.cpp-src -B llama.cpp-src/build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build llama.cpp-src/build \
  --target llama-kalm-reranker -j

For CPU-only use, configure with -DGGML_CUDA=OFF.

Score a pair

llama.cpp-src/build/bin/llama-kalm-reranker \
  -m kalm-reranker-v1-small-q4_k_m.gguf -ngl 99 --require-gpu \
  --query "What is the capital of China?" \
  --passage "The capital of China is Beijing."

The output includes yes_logit, no_logit, margin, and score = sigmoid(yes_logit - no_logit). Rank larger margins first. For JSONL and reproducible examples, see examples/.

Complete FIQA evaluation

The evaluation covers all 648 FIQA test queries and reranks the frozen retriever top-100: 64,800 scored pairs.

The Q4_K_M full-set fidelity gate accepts at most 0.020 drop from BF16 for both NDCG@10 and MRR@10. Observed drops were 0.00047 and 0.00017 respectively; the gate status is accepted. Throughput in the recorded run was 26.634 pairs/s. Hardware and scheduling affect throughput.

See EVALUATION.md for methodology and full provenance.

Runtime boundaries

• Single query/passage sequence and sequential JSONL scoring.
• Encoder limit 1,024 tokens; query budget 512 tokens.
• Encoder chunk size 4.
• No KV cache, Flash Attention, generation, or server mode.

Troubleshooting

• An unknown t5gemma2 architecture means the seven patches were not applied to the pinned upstream commit.
• A --require-gpu failure means full CUDA offload was not achieved.
• Use the CPU build without --require-gpu when CUDA is unavailable.
• Verify the model with sha256sum --check SHA256SUMS.

License

Apache-2.0. See LICENSE and THIRD_PARTY_NOTICES.md.

Citation

If you find this model useful, please consider citing our papers.

@misc{zhao2026kalmrerankerv1,
      title={KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking}, 
      author={Xinping Zhao and Jiaxin Xu and Ziqi Dai and Xin Zhang and Shouzheng Huang and Danyu Tang and Xinshuo Hu and Meishan Zhang and Baotian Hu and Min Zhang},
      year={2026},
      eprint={2606.22807},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.22807}, 
}

@misc{zhao2026kalmembeddingv2,
      title={KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model}, 
      author={Xinping Zhao and Xinshuo Hu and Zifei Shan and Shouzheng Huang and Yao Zhou and Xin Zhang and Zetian Sun and Zhenyu Liu and Dongfang Li and Xinyuan Wei and Youcheng Pan and Yang Xiang and Meishan Zhang and Haofen Wang and Jun Yu and Baotian Hu and Min Zhang},
      year={2025},
      eprint={2506.20923},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.20923}, 
}

@misc{hu2025kalmembedding,
      title={KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model}, 
      author={Xinshuo Hu and Zifei Shan and Xinping Zhao and Zetian Sun and Zhenyu Liu and Dongfang Li and Shaolin Ye and Xinyuan Wei and Qian Chen and Baotian Hu and Haofen Wang and Jun Yu and Min Zhang},
      year={2025},
      eprint={2501.01028},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.01028}, 
}