加载中…
加载中…
Core ML conversion of IBM's ibm-granite/granite-embedding-30m-sparse
(revision ad82b1fd), the learned sparse encoder behind Intelligent Internet's
Evoke. It turns text into a short list of weighted vocabulary terms,
including related terms the text never uses ("vaccines for seniors" → vaccine, vaccination, older, elder, …), which go
into an ordinary inverted index next to BM25. 30M parameters, fp16, Apache-2.0. IBM authored the model; Fluid Inference
converted it.
| Package | Input | Output | Size |
|---|---|---|---|
granite-embedding-30m-sparse-L64-fp16.mlpackage |
input_ids int32 [1, 64] |
max_logits fp16 [1, 192], vocab_ids int32 [1, 192] |
58 MB |
granite-embedding-30m-sparse-L128-fp16.mlpackage |
[1, 128] |
same | 58 MB |
granite-embedding-30m-sparse-L256-fp16.mlpackage |
[1, 256] |
same | 58 MB |
granite-embedding-30m-sparse-L512-fp16.mlpackage |
[1, 512] |
same | 58 MB |
One package per sequence length (fixed shapes keep the graph on the Neural Engine; enumerated shapes ran entirely on CPU). macOS 14 / iOS 17 or newer.
tokenizer.json), <s> … </s>, right-padded with <pad> (id 1) to the
package length. The attention mask is derived from <pad> inside the graph.w = log1p(max(v, 0)) ^ gamma × scale over the first active_dims, keep w > 0.
config.json holds Evoke P2.2's constants: queries keep 50 terms (gamma 1.8519, scale 0.6964), documents keep 192
(gamma 0.5628, scale 1.0). Score = Σ query weight × document weight over shared terms.Against Evoke's shipped ONNX compilers (Intelligent-Internet/Evoke-Model-Beta-1), NFCorpus test (323 queries,
3,633 documents), sparse dot over the semantic terms only:
| Encoder | nDCG@10 | Recall@100 | Recall@1000 | Same top-10 as ONNX |
|---|---|---|---|---|
| ONNX (shipped, CPU) | 0.3402 | 0.2956 | 0.5854 | — |
| Core ML fp16, GPU | 0.3405 | 0.2957 | 0.5856 | 99.6% |
| Core ML fp16, Neural Engine | 0.3405 | 0.2953 | 0.5847 | 98.7% |
An fp32 build (not uploaded) reproduces the ONNX term sets exactly (200/200 queries and documents); the fp16 differences are low-weight terms at the top-k boundary.
Apple M5 Pro, macOS 27, one warm call (median of 100), Swift:
| Length | Neural Engine | GPU | CPU |
|---|---|---|---|
| 64 | 0.74–0.87 ms | 2.5–3.3 ms | 2.8–3.1 ms |
| 128 | 1.2–1.4 ms | 1.5–4.3 ms | 4.1 ms |
| 256 | 2.9 ms | 1.9–3.4 ms | 9.5–15 ms |
| 512 | 7.6 ms | 2.6 ms | 16–27 ms |
Use the Neural Engine up to 128 tokens and the GPU beyond. 160 of 172 ops run on the Neural Engine; token lookup, mask setup and the final top-k stay on CPU. After several seconds idle the first Neural Engine call takes ~15 ms. Encoding 3,633 NFCorpus documents: 14.1 s Core ML GPU vs 88.2 s ONNX Runtime CPU.
FluidUse wraps the packages (EvokeManager, Swift RoBERTa tokenizer with token ids identical to the Python tokenizer,
host-side weighting; PR #28) and includes EvokeSearchDemo, an
auto-playing search-as-you-type demo.