SauerkrautLM-Multi-Reason-ModernColBERT-GGUF (ColBERT GGUF)

This repository contains GGUF format weights for the ColBERT retrieval model VAGOsolutions/SauerkrautLM-Multi-Reason-ModernColBERT, converted specifically for our GGML-based ColBERT C++ runtime.

This GGUF file is NOT a standard llama.cpp decoder model. It uses the pg_colbert_v1 layout containing backbone transformer weights, tokenizer metadata, ColBERT dense projection layers, and similarity metrics.

Model Summary

Source Model: VAGOsolutions/SauerkrautLM-Multi-Reason-ModernColBERT
Format: GGUF (pg_colbert_v1 schema)
Precision: F16
Encoder Backbone: ModernBERT / BERT
ColBERT Projection Dimension: 128
Profile Schema: pg_colbert_profile_v1
Target Runtime: both

Parity & Verification Report

Strict PyLate Token-Plan Parity: PASSED
Vector Parity Checked: YES
Strict Vector Parity Status: PASSED
Vector Parity Claim: PASSED (verified numerical equivalence with reference PyLate embeddings)

Usage with ColBERT GGML C++ Runtime

This GGUF model is designed to be loaded directly by our custom C++ ColBERT runtime (which uses GGML for the ModernBERT/BERT backbone forward pass and executes the ColBERT late-interaction dense projection).

Loading the Model in C++

// Initialize the ColBERT GGML model context
colbert_model model = colbert_model_load("SauerkrautLM-Multi-Reason-ModernColBERT.f16.gguf");

// Tokenize and encode queries into late-interaction token embeddings
std::vector<float> query_embeddings = colbert_encode_query(model, "Which planet is known as the Red Planet?");

// Tokenize and encode documents into late-interaction token embeddings
std::vector<float> doc_embeddings = colbert_encode_doc(model, "Mars is the Red Planet.");

GGUF Conversion Info

Generated using the convert_colbert_hf_to_gguf.py utility.