SauerkrautLM-MoEdernColBERT-27M-ONNX

ONNX exports of VAGOsolutions/SauerkrautLM-MoEdernColBERT-27M at revision d2b7b1feaaac1bf87510ca9525208c3daeabcc5a, for CPU inference with ONNX Runtime in Python, Node.js and browsers (WASM). Each graph turns token IDs into L2-normalized 128-dimensional token vectors. ColBERT MaxSim over those vectors scores a query against a passage.

Files

manifest.json lists every file with its size and SHA-256. runtime.json holds the tokenization contract: prefixes, lengths, padding, the punctuation tokens excluded from document scoring. parity-report.json holds every measurement summarized below.

• exact graphs give the same token vectors in Python, Node.js and browser ONNX Runtime, within 0.0001 per value and 0.001 per MaxSim score.
• quality graphs depend on runtime kernels, so their vectors differ between runtimes. Compare them only with vectors from the same runtime, and key caches by runtime.

Verification

moedern-onnx parity measured these values on macOS-26.7.1-arm64-arm-64bit with Python 3.11.17, ONNX Runtime 1.31.1 and PyTorch 2.6.0. The FP32 graph matches the upstream Sentence Transformers model within 8.6e-07 per value, across tokenizer edge cases and batch shapes up to 3 × 512 tokens.

The other precisions are compared with FP32 on a retrieval fixture of 24 hand-written queries over 24 passages in seven languages. The fixture is a regression check, not a retrieval benchmark.

Inputs and outputs

• Inputs input_ids and attention_mask: int64, shape [batch, sequence], at most 512 positions.
• Output token_embeddings: float32, shape [batch, sequence, 128], unit length per token.
• Queries start with [Q] and always use 32 positions: the tokenizer's padding ([MASK]) fills the rest, and every position counts in MaxSim.
• Documents start with [D] and use at most 512 positions. Padding and punctuation tokens are excluded from MaxSim.

Usage in Python

Needs onnxruntime, tokenizers and numpy.

from pathlib import Path
import json

import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

folder = Path("SauerkrautLM-MoEdernColBERT-27M-ONNX")  # hf download kyr0/SauerkrautLM-MoEdernColBERT-27M-ONNX --local-dir SauerkrautLM-MoEdernColBERT-27M-ONNX
config = json.loads((folder / "runtime.json").read_text())
tokenizer = Tokenizer.from_file(str(folder / "tokenizer.json"))
tokenizer.no_padding()
tokenizer.no_truncation()
session = ort.InferenceSession(str(folder / "model_q8.onnx"))


def encode(text, kind):
    ids = tokenizer.encode(config["prefixes"][kind] + text).ids
    limit = config["query_length"] if kind == "query" else config["max_document_length"]
    if len(ids) > limit:
        raise ValueError(f"{kind} has more than {limit} tokens; chunk it first")
    width = limit if kind == "query" else len(ids)
    input_ids = np.full((1, width), config["pad_token_id"], dtype=np.int64)
    input_ids[0, : len(ids)] = ids
    mask = (np.arange(width) < len(ids)).astype(np.int64)[None]
    vectors = session.run(None, {"input_ids": input_ids, "attention_mask": mask})[0][0]
    if kind == "query":
        return vectors  # all 32 positions count, including the [MASK] expansion
    return vectors[~np.isin(input_ids[0], config["skip_token_ids"])]


def maxsim(query, document):
    return float((query @ document.T).max(axis=1).sum())


query = encode("Wie hoch ist die Zugspitze?", "query")
passages = ["Die Zugspitze ist 2962 Meter hoch.", "Paris is the capital of France."]
scores = [maxsim(query, encode(passage, "document")) for passage in passages]
print(sorted(zip(scores, passages), reverse=True))

Conversion

The exporter re-implements the checkpoint's two attention and mixture-of-experts layers with plain tensor operations and exports them with torch.onnx.export (opset 17, dynamic batch and sequence axes). The graph includes the projection head and per-token normalization.

Changes from the upstream model, as required by the Apache License 2.0:

1. Every routed expert is computed and the learned router gates weight their outputs. Results match upstream, but the graph does not skip unused experts.
2. Positions are capped at 512; upstream's sliding-window attention for longer inputs is not exported.
3. Quantized variants store rounded weights (see the files table); the router stays FP32 because small router errors can change which experts run.

Limitations

• Shipped precisions: fp32, fp16, q8, int8, q4.
• q8 makes the download smaller; it does not make inference faster or reduce memory.
• int8 quantizes activations per batch at run time, so a passage's vectors depend slightly on the other passages in its batch.

License

Apache-2.0, the license of the upstream checkpoint. These graphs are derivatives of VAGOsolutions/SauerkrautLM-MoEdernColBERT-27M.