SauerkrautLM-MoEdernColBERT-1.5M-CoreML

Core ML packages of SauerkrautLM-MoEdernColBERT-1.5M for Apple GPUs and the Neural Engine. The 4-bit package is 0.87 MB and scores 0.5903 on NanoBEIR English (fp32: 0.5912); a query takes 0.13 ms (pal4, Neural Engine) on an Apple M5 Max.

The SauerkrautLM-MoEdernColBERT family introduces MoEdernBERT, a new ModernBERT-style encoder architecture with a mixture of experts and hash embeddings. MoEdernColBERT-1.5M has 1.47M parameters, of which only 0.88M are active per token.

Variants

NanoBEIR English: mean NDCG@10 over 13 datasets, sentence-transformers 6 evaluator with the model's own configuration, document package on the GPU; on the Neural Engine the result differs by at most 0.001. Size: the 16 × 512 document package (the other two packages are about the same size). Speed: Apple M5 Max, macOS 26.3.2. Documents/s: batches of 16 documents × 512 tokens, median of three runs, faster of CPU_AND_GPU and ALL (Core ML runs the document package on the GPU in both). Latency: p50 of 300 runs from token ids to the embedding; the last column embeds one query and one 256-token document and computes MaxSim, on the fastest compute unit.

Every variant stays within 0.004 NDCG@10 of the fp32 model; small gains over fp32 are quantization noise.

• int8 — the default choice: half the size of fp16 with practically the same embeddings.
• pal4 — the smallest: weights palettized to 4 bits (k-means, groups of 16 channels).
• fp16 — highest fidelity.

All variants compute in fp16 on the device; int8 and pal4 only compress the stored weights, the router stays in fp16.

Packages

Every variant comes as three packages with fixed input shapes:

Inputs: input_ids and attention_mask, both int32. Output: embeddings, float32 of shape (batch, length, 128), already passed through the ColBERT head and L2-normalized. The GPU is 5 to 7 times faster than the Neural Engine for document batches; for single queries, the Neural Engine is the fastest or close to it.

Usage

pip install -U huggingface_hub coremltools numpy tokenizers
hf download VAGOsolutions/SauerkrautLM-MoEdernColBERT-1.5M-CoreML --local-dir moederncolbert-1.5m-coreml \
  --include "*-int8-*" --include "*.py" --include "tokenizer.json"
cd moederncolbert-1.5m-coreml

from moedern_coreml import MoEdernColBERTCoreML

model = MoEdernColBERTCoreML(
    query_package="SauerkrautLM-MoEdernColBERT-1.5M-int8-query-1x32.mlpackage",      # Neural Engine
    document_package="SauerkrautLM-MoEdernColBERT-1.5M-int8-doc-16x512.mlpackage",   # GPU
)

queries = ["Wie hoch ist die Zugspitze?", "What is the capital of France?"]
documents = [
    "Die Zugspitze ist mit 2962 Metern der höchste Berg Deutschlands.",
    "Paris is the capital and largest city of France.",
    "Berlin ist die Hauptstadt Deutschlands.",
]

query_embeddings = model.encode_queries(queries)          # list of (32, 128) arrays
document_embeddings = model.encode_documents(documents)  # list of (num_tokens, 128) arrays

scores = model.scores(query_embeddings, document_embeddings)  # MaxSim, shape (2, 3)
print(scores.argmax(axis=1))  # [0 1]

python example.py runs the same code. Requires macOS 15 or later; we tested on macOS 26.3.2 on an Apple M5 Max. The first load compiles each package for the device, which can take a while for the larger packages; later loads are fast.

Pre- and post-processing

moedern_coreml.py does this for you. If you call the packages yourself:

• Tokenizer: tokenizer.json from this repository (the same as in the main model). Token ids: [CLS] 2, [SEP] 3, [MASK] 4, [Q] 30000, [D] 30001.
• Queries: tokenize "[Q] " + query to [CLS] [Q] … [SEP], truncate to 32 tokens and pad to exactly 32 tokens with [MASK]. Use an attention_mask of all ones; the model attends over the [MASK] expansion. Keep all 32 output vectors.
• Documents: tokenize "[D] " + document to [CLS] [D] … [SEP], truncate to the package length (512 or 256) and pad with [MASK] and attention_mask 0. Fill unused rows of a batch with [CLS] [SEP] (attention_mask 1 on these two tokens). Afterwards drop the padding and the vectors of punctuation tokens (the characters of Python's string.punctuation that are single tokens in the vocabulary).
• Score: MaxSim. For each query vector take the highest dot product with any document vector, then sum over the query vectors.

How the packages were built

From a static re-implementation of the model with coremltools 9.0. All experts are computed densely and weighted by the router, which gives the same result as top-2 routing with fixed tensor shapes. Norms are computed in a scaled form, so fp16 does not overflow (in PyTorch, .half() does). Attention subtracts the row maximum before the softmax, which avoids wrong results from the Core ML GPU backend on macOS 26.

Limitations

• Documents up to 512 tokens (256 with the doc-1x256 package). The ±256-token window for documents of up to 32k tokens is part of the main model only.
• Apple platforms only. For Linux and Windows CPUs, use the GGUF version; on NVIDIA GPUs, the main model with its bundled Triton kernel.
• Languages: German, English, Spanish, Dutch, French, Italian and Portuguese. Arabic, Japanese and Korean are not supported.

License

Apache 2.0, see LICENSE.

Citation

@misc{sauerkrautlm_moederncolbert_2026,
  title={MoEdernBERT: Hash-Embedded Mixture-of-Experts Encoders for Tiny Multilingual Late-Interaction Retrieval},
  author={David Golchinfar},
  organization={VAGO Solutions},
  url={https://huggingface.co/VAGOsolutions/SauerkrautLM-MoEdernColBERT-1.5M},
  year={2026}
}

Contact

VAGO solutions