SauerkrautLM-MoEdernColBERT-27M-GGUF

GGUF files of SauerkrautLM-MoEdernColBERT-27M for CPU inference with our small ggml-based engine. The 4-bit file is 18.9 MB and scores 0.6152 on NanoBEIR English (fp32: 0.6192).

The SauerkrautLM-MoEdernColBERT family introduces MoEdernBERT, a new ModernBERT-style encoder architecture with a mixture of experts and hash embeddings. MoEdernColBERT-27M has 27.0M parameters, of which only 12.3M are active per token.

Note: These files need the engine in this repository. Stock llama.cpp, Ollama and LM Studio cannot load them.

Files

NanoBEIR English: mean NDCG@10 over 13 datasets, sentence-transformers 6 evaluator with the model's own configuration. Speed: Apple M5 Max (18-core CPU), macOS 26.3.2. Documents/s: batches of 16 documents × 512 tokens, median of three runs, fastest of 6, 12 and 18 threads. Query latency: p50 of 300 runs from token ids to the query embedding (32 tokens), faster of 6 and 12 threads.

Every file stays within 0.004 NDCG@10 of the fp32 model; small gains over fp32 are quantization noise.

• Q8_0 — the default choice: practically lossless and about as fast as Q4_0.
• Q4_0 — the smallest file: attention and expert matrices in 4 bit, ColBERT head in 8 bit.
• F16 — near-exact.
• F32 — the same embeddings as the PyTorch model up to float rounding; useful as a reference when porting the engine.

In Q8_0 and Q4_0, the hash embedding stays F16, and the router and all norms stay F32.

Usage

1. Download and build the engine

pip install -U huggingface_hub numpy tokenizers
hf download VAGOsolutions/SauerkrautLM-MoEdernColBERT-27M-GGUF --local-dir moederncolbert-27m-gguf \
  --include "*Q8_0.gguf" --include "*.py" --include "tokenizer.json" --include "engine/*"
cd moederncolbert-27m-gguf

git clone https://github.com/ggml-org/ggml ggml_src
git -C ggml_src checkout 3612c0af93f2cfc8f048295300a61187723b12ae
cmake -S engine -B engine/build -DCMAKE_BUILD_TYPE=Release
cmake --build engine/build -j

You need CMake 3.14 or newer and a C/C++ compiler. The engine is a single C file (engine/main_v10.c) on top of ggml's CPU backend. We built and tested it on macOS with Apple Silicon.

2. Encode and score

from moedern_gguf import MoEdernColBERTGGUF

with MoEdernColBERTGGUF("SauerkrautLM-MoEdernColBERT-27M-Q8_0.gguf") as model:
    queries = ["Wie hoch ist die Zugspitze?", "What is the capital of France?"]
    documents = [
        "Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.",
        "Paris is the capital and largest city of France.",
        "Berlin ist die Hauptstadt Deutschlands.",
    ]

    query_embeddings = model.encode_queries(queries)          # list of (32, 128) arrays
    document_embeddings = model.encode_documents(documents)  # list of (num_tokens, 128) arrays

    scores = model.scores(query_embeddings, document_embeddings)  # MaxSim, shape (2, 3)
    print(scores.argmax(axis=1))  # [0 1]

python example.py runs the same code. moedern_gguf.py starts the engine as a child process and handles tokenization, padding and the skiplist. Pass threads= to set the CPU threads: about the number of performance cores works best, all cores is usually slower.

Pre- and post-processing

If you call the engine yourself, prepare the input exactly like this:

• Tokenizer: tokenizer.json from this repository (the same as in the main model). Token ids: [CLS] 2, [SEP] 3, [MASK] 4, [Q] 30000, [D] 30001.
• Queries: tokenize "[Q] " + query to [CLS] [Q] … [SEP], truncate to 32 tokens and pad to exactly 32 tokens with [MASK]. Send all 32 tokens; the model attends over the [MASK] expansion. Keep all 32 output vectors.
• Documents: tokenize "[D] " + document to [CLS] [D] … [SEP] and truncate to 512 tokens. Send only the real tokens, no padding. Afterwards drop the vectors of punctuation tokens (the characters of Python's string.punctuation that are single tokens in the vocabulary).
• Output: one 128-dimensional, L2-normalized vector per token. The ColBERT head is already applied.
• Score: MaxSim. For each query vector take the highest dot product with any document vector, then sum over the query vectors.

Engine protocol

engine/build/ggml_infer_v10 --model <file.gguf> --serve --threads <n> reads requests from stdin and writes results to stdout, all values little-endian:

• Request: uint32 magic 0x51524747, request id, total number of tokens, flags 2; then uint32 number of sequences and one uint32 length per sequence; then all token ids as int32, sequence after sequence.
• Response: uint32 magic 0x50524747, request id, status (0 = ok), number of tokens, dimension (128); then one row of float32 values per input token.

At most 1,024 sequences and 65,536 tokens per request. moedern_gguf.py is a complete reference client.

Rebuilding the files

engine/convert_v10_to_gguf.py --checkpoint-dir <download of SauerkrautLM-MoEdernColBERT-27M> --outfile model.gguf --outtype q8_0 rebuilds every file of this repository from the main model with identical tensors (needs numpy, safetensors and gguf==0.19.0; --outtype is one of f32, f16, q8_0, q4_0).

Limitations

• CPU only. On NVIDIA GPUs, the main model with its bundled Triton kernel is much faster; on Apple devices, use the CoreML version.
• Documents up to 512 tokens. The ±256-token window for documents of up to 32k tokens is part of the PyTorch model only; the engine has no such window, so documents are cut at 512 tokens.
• Languages: German, English, Spanish, Dutch, French, Italian and Portuguese. Arabic, Japanese and Korean are not supported.

License

Apache 2.0, see LICENSE. This covers the model files and the code in this repository. The engine links against ggml (MIT License, © the ggml authors), which the build fetches from GitHub; ggml itself is not included here.

Citation

@misc{sauerkrautlm_moederncolbert_2026,
  title={MoEdernBERT: Hash-Embedded Mixture-of-Experts Encoders for Tiny Multilingual Late-Interaction Retrieval},
  author={David Golchinfar},
  organization={VAGO Solutions},
  url={https://huggingface.co/VAGOsolutions/SauerkrautLM-MoEdernColBERT-27M},
  year={2026}
}

Contact

VAGO solutions