Granite Embedding 97M Multilingual R2, W8A16 for IREE

Unofficial. This is a weight-only INT8 quantization of IBM's ibm-granite/granite-embedding-97m-multilingual-r2, compiled into IREE modules. It was not made, reviewed or endorsed by IBM. All credit for the model goes to IBM's Granite Embedding team; problems with this conversion are ours. Upstream is licensed under Apache 2.0, and so is this derivative (see LICENSE).

• Upstream: ibm-granite/granite-embedding-97m-multilingual-r2, revision 835ad14087e140460703cf0fae09f97d469d65c2
• Model id: granite-embedding-97m-multilingual-r2@835ad140+w8a16-int8sym-perchannel
• IREE: 3.13.0 (iree-base-compiler==3.13.0). A .vmfb loads only in an IREE runtime of the same version.
• Output: 384-dim FP32, L2-normalized (cosine similarity = dot product)

Download

hf download edvv1n/granite-embedding-97m-multilingual-r2-w8a16-iree
modelscope download --model LumilioPhotos/granite-embedding-97m-multilingual-r2-w8a16-iree

A runtime needs manifest.json, tokenizer.json and the one .vmfb for its device; manifest.json gives each file's sha256.

What was changed (Apache 2.0, section 4)

Relative to the upstream files at the revision above:

1. model.safetensors: the token embedding and every Linear weight are quantized to INT8 with FP32 scales (rules below); the LayerNorm weights are converted from BF16 to FP32 (exact). Tensor X.weight becomes X.weight_int8 (I8, [out, in]) and X.weight_scale (F32, [out]).
2. The forward pass of transformers' ModernBertModel + CLS pooling + L2 normalization (upstream's sentence-transformers pipeline) was re-written in PyTorch with the precision steps below, exported with iree-turbine and compiled with iree-compile into the .vmfb files under iree/.
3. Added: manifest.json, quantization_config.json, reference/vectors.json, this README, and LICENSE (the Apache 2.0 text; upstream declares the license in its model card and ships no LICENSE or NOTICE file). Not included: upstream's README, ONNX, OpenVINO and sentence-transformers files and its model.sig signature (it signs upstream's files, not these).

config.json, tokenizer.json, tokenizer_config.json and special_tokens_map.json are upstream's, unchanged. model.safetensors here is the input to the IREE export, so modules for another IREE version can be rebuilt from it; transformers cannot load it as is.

Precision

Signed INT8 in [-127, 127], zero point 0, FP32 scales, no groups, no activation quantization, no integer GEMM:

s_c = max(|W_c|) / 127        W_q,c = round(W_c / s_c)    (round half to even)

The INT8 constants stay INT8 in the modules (each .vmfb is about the size of model.safetensors). Dequantization is explicit in the graph: on every call each Linear weight is dequantized (int8 -> f32 * s_c -> f16, transposed to [in, out]) by its own dispatch, and the FP16 matmul reads the result (IREE accumulates FP16 matmuls in FP32 and rounds the output to FP16). The token embedding gathers INT8 rows and their scales, then dequantizes only those rows. The modules are compiled with --iree-opt-const-eval=false --iree-opt-const-expr-hoisting=false so the compiler does not fold the weights back into FP16 constants.

Entry points

Shapes are static. Every .vmfb exports the same functions; they share one copy of the weights.

Signature of each (B, S from the table):

input_ids:      i64 [B, S]   tokenizer.json ids, right-padded with 179935 (<|endoftext|>)
attention_mask: i64 [B, S]   1 for tokens, 0 for padding
-> embedding:   f32 [B, 384] L2-normalized CLS embedding

Tokenize with tokenizer.json (it adds <|startoftext|> ... <|return|>; the first token is the CLS token that is pooled), truncate to S, pad to S. Rows that are all padding (to fill a batch) return finite vectors to discard. Padding does not change a sentence's vector beyond float rounding.

Files and targets

Both modules were run and checked on this release's build machine (Apple M2 Pro, macOS 27.2); see Verification. manifest.json lists every file with its size, sha256, target, compile flags and function signatures.

CUDA, Vulkan and x86_64 CPU modules can be compiled from the same quantized checkpoint with the same IREE version, but they are not published in this release.

Verification

51 synthetic sentences written for this check (Chinese, English, mixed Chinese/English, single terms, definitions, unrelated distractors, code and five passages of 313-487 tokens), with 18 cross-language concept pairs such as 特征值 / eigenvalue. Reference: the upstream model in FP32 (transformers 5.22.2, eager attention, each sentence unpadded), CLS pooling, L2 norm. Each row runs one entry point over every sentence that fits it.

• cos: cosine to the reference, per sentence
• top-5: overlap of each sentence's 5 nearest neighbours (among the sentences in that row) with the reference's
• top-1: share of sentences whose nearest neighbour matches the reference's
• pairs: cross-language pair retrieval hits (each side of a pair retrieves the other among all pair members of the other language); the reference's score is in parentheses

torch-fp32 is the re-implemented forward pass with upstream's weights in FP32 (it must match transformers to cosine >= 0.99999), and torch-w8a16 the quantized graph in PyTorch (CPU, FP16); both for comparison. Between every verified target and entry point, the same sentence gives cosine >= 0.999999 (threshold 0.99995).

Pass thresholds: cosine min >= 0.999, top-5 overlap mean >= 0.95, top-1 agreement >= 0.95, pair retrieval answers identical to the reference, cross-target cosine >= 0.99995.

reference/vectors.json holds these sentences with their token ids, the reference vectors and the W8A16 vectors, so an installation can check itself.

Latency

iree-benchmark-module (median of 5), Apple M2 Pro, macOS 27.2:

Limitations

• Static shapes only: texts longer than 512 tokens must be split by the caller; upstream supports 32768. Short texts pay for the whole bucket.
• Quantization moves vectors slightly (cosine to FP32 in the table above), so do not mix these vectors with upstream FP32 vectors in one index.
• Only Apple silicon Macs (CPU and Metal) are covered by this release.
• The tokenizer's own padding and truncation settings in tokenizer.json (padding to the longest, max 32768) are upstream's; set them to the entry point's shape.
• Everything upstream says about intended use, bias and risks applies here.