SauerkrautLM-MoEdernColBERT-115M

The flagship of the family: a 115M-parameter multilingual ColBERT retriever at a fifth of the size of Qwen3-Embedding-0.6B and 8× as fast. Built for German and six other European languages, with documents of up to 32k tokens.

The SauerkrautLM-MoEdernColBERT family introduces MoEdernBERT, a new ModernBERT-style encoder architecture with a mixture of experts and hash embeddings. MoEdernColBERT-115M has 114.6M parameters, of which only 67.4M are active per token.

Highlights

• Ahead of 2.7× larger granite-311m-multilingual-r2 on all 11 benchmarks in the overview.
• 2.5× the throughput of granite-311m-multilingual-r2 and 8× that of Qwen3-Embedding-0.6B: 1,132 documents per second on one GPU.
• German retrieval ahead of models up to 5× larger. MKQA de→en 0.731 vs. 0.671 for Qwen3-Embedding-0.6B.
• Dutch retrieval ahead of models up to 5× larger. NFCorpus-NL 0.340 vs. 0.283† for Qwen3-Embedding-0.6B.
• German legal texts: ahead of 8B models and commercial APIs. LegalQuAD 0.851 vs. 0.772† for Nemotron-3-Embed-8B.
• Multilingual quality of 0.6B models at 1/5 the size. NanoBEIR in six languages 0.585, above Qwen3-Embedding-0.6B and jina-embeddings-v3.
• Long documents up to 32k tokens. LongEmbed 90.1, ahead of the 7B–14B models on the public LongEmbed leaderboard.

† public MTEB result, not measured by us.

Architecture: MoEdernBERT

6 gated convolution layers and 4 transformer layers, hidden size 1024.

• Hash embedding. A factorized embedding: every token id owns a short row of 192 values, which a shared layer projects to the hidden size. This needs 5.96M parameters instead of 30.7M.
• Mixture-of-experts MLP. In every transformer layer, a router sends each token to 2 of 8 GeGLU experts, and one shared expert always runs.
• Gated short convolutions. Six of the ten layers are cheap depthwise convolutions (kernel 3) for local context.
• Attention and long context. Global attention with RoPE, trained on 512 tokens. Above 513 tokens, a ±256-token sliding window keeps every token within the trained range, so documents of up to 32k tokens work.
• ColBERT head. Every token is mapped to 128 dimensions. Queries are padded to 32 tokens with [MASK], and relevance is scored with MaxSim.

Training: Stella-style embedding distillation, then knowledge-distillation pre-training with a final annealing phase.

Speed

Sparse activation makes MoEdernColBERT-115M fast on a GPU and on a laptop.

One NVIDIA RTX 6000 Ada, bf16, 512-token documents, bundled Triton MoE kernel.

Results

German

Other languages

German legal texts

Long documents

The MoEdernColBERT family

NDCG@10 unless noted, higher is better. Throughput measured as under Speed.

Usage

The model ships its own modeling code, so trust_remote_code=True is required. Tested with sentence-transformers 6.1 and transformers 5.

pip install -U "sentence-transformers>=6.1"
pip install -U triton   # Linux + NVIDIA GPU: enables the bundled fused MoE kernel

Fast GPU path. On NVIDIA GPUs, the bundled Triton MoE kernel switches on automatically during inference; without a CUDA GPU or without Triton, the model falls back to plain PyTorch. Set MOEDERN_TRITON=0 to turn the kernel off.

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-MoEdernColBERT-115M", trust_remote_code=True)

queries = ["Wie hoch ist die Zugspitze?", "What is the capital of France?"]
documents = [
    "Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.",
    "Paris is the capital and largest city of France.",
    "Berlin ist die Hauptstadt Deutschlands.",
]

query_embeddings = model.encode_query(queries)          # list of (32, 128) tensors
document_embeddings = model.encode_document(documents)  # list of (num_tokens, 128) tensors

scores = model.similarity(query_embeddings, document_embeddings)  # MaxSim, shape (2, 3)
print(scores.argmax(dim=1))  # tensor([0, 1])

Long documents. Documents are cut at 512 tokens by default. Raise the limit, and the ±256 window switches on automatically:

model[0].document_length = 32768

Precision. Use fp32 or bf16. Do not call .half(): fp16 overflows and produces NaN embeddings.

GGUF and CoreML. Quantized versions run on CPUs with our ggml-based engine (GGUF) and on Apple GPUs and the Neural Engine (CoreML). The 4-bit CoreML file is 57.9 MB and scores 0.642 on NanoBEIR English (fp32: 0.643).

Evaluation

All numbers without † come from our own runs with sentence-transformers 6, each model with its own configuration. On the German tasks, our models use the better of 512 or 8,192 document tokens per task. The paper has the full protocol and all results.

Languages and limitations. German, English, Spanish, Dutch, French, Italian and Portuguese. Arabic, Japanese and Korean are not supported.

License

Apache 2.0, see LICENSE.

Citation

@misc{sauerkrautlm_moederncolbert_2026,
  title={MoEdernBERT: Hash-Embedded Mixture-of-Experts Encoders for Tiny Multilingual Late-Interaction Retrieval},
  author={David Golchinfar},
  organization={VAGO Solutions},
  url={https://huggingface.co/VAGOsolutions/SauerkrautLM-MoEdernColBERT-115M},
  year={2026}
}

Contact

VAGO solutions