papr-embed-code-1-0.6b, a code-search embedding model, converted to CoreML so it runs on the Apple Neural Engine. A search query embeds in 17 ms on an M2 Pro, and the vectors match the original PyTorch model to cosine 0.9998.
Use it to give an on-device coding agent or editor semantic code search without a GPU, a server or network access. The vectors are interchangeable with the PyTorch, sentence-transformers and ONNX builds, so you can index in the cloud and search on device, or the other way around.
| Base model | papr-ai/papr-embed-code-1-0.6b (Qwen3-Embedding-0.6B, fine-tuned for code search) |
| Output | 1024-d, L2-normalized (pooling and normalization happen inside the model) |
| Precision | fp16 weights and compute, 1.2 GB |
| Lengths | three fixed-length functions: len_64, len_256, len_1024 |
| Requires | Apple Silicon (M1 or later), macOS 15+ (multifunction models) |
Measured on a MacBook Pro M2 Pro (16 GB), macOS 15.6, ComputeUnit.CPU_AND_NE. Cosine is against PyTorch fp32 on the same token ids. Lowest cosine over the samples shown.
| Function | Use for | Latency p50 | p95 | Cosine vs fp32 (min) |
|---|---|---|---|---|
len_64 |
search queries | 16.7 ms | 17 ms | 0.9998 |
len_256 |
short files, snippets | 45 ms | 50 ms | 0.9997 |
len_1024 |
files up to ~1k tokens | 306 ms | 309 ms | 0.9997 |
For comparison, on the same Mac the int8 ONNX build on CPU takes about 350 ms per query (cosine 0.9998). 2,079 of the 2,083 ops run on the Neural Engine; the other 4 are the embedding lookup. Raw numbers are in results/coreml_parity.json.
pip install coremltools tokenizers huggingface_hub numpy
import numpy as np, coremltools as ct
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer
d = snapshot_download("papr-ai/papr-embed-code-1-0.6b-CoreML", local_dir="papr-embed-code-1-0.6b-CoreML")
tok = Tokenizer.from_file(f"{d}/tokenizer.json")
m = ct.models.MLModel(f"{d}/papr-embed-code-1-0.6b.mlpackage",
compute_units=ct.ComputeUnit.CPU_AND_NE, function_name="len_64")
INSTRUCT = "Given a code search query about a repository, retrieve the source code file that must be read or edited"
ids = tok.encode(f"Instruct: {INSTRUCT}\nQuery:retry an upload with backoff", add_special_tokens=False).ids + [151643]
x = np.zeros((1, 64), np.int32); x[0, :len(ids)] = ids
v = m.predict({"input_ids": x, "last_index": np.array([len(ids) - 1], np.int32)})["embedding"][0] # unit vector
examples/embed.py handles all three lengths, queries and documents.
| Input | Type | Value |
|---|---|---|
input_ids |
int32 [1, L] |
token ids, `< |
last_index |
int32 [1] |
index of the `< |
Output: embedding, float [1, 1024], already unit-length. Cosine similarity is a dot product.
L fits your token count plus one.Instruct: <task>\nQuery:<text>. Documents (code files, chunks) are embedded bare. Any task sentence works, but use the same one for indexing and search tests. The one above is what we evaluate with.Swift:
let cfg = MLModelConfiguration()
cfg.computeUnits = .cpuAndNeuralEngine
cfg.functionName = "len_64"
let model = try MLModel(contentsOf: compiledURL, configuration: cfg) // compile once with MLModel.compileModel(at:)
<|endoftext|> (151643). With recent transformers, tokenizer.eos_token resolves to <|im_end|> (151645). Those vectors look normal but land at cosine 0.5–0.8 from the real ones, and retrieval quietly gets worse.local_dir= (or hf download ... --local-dir). CoreML can't compile an .mlpackage through the symlinks in the default cache and fails with weight.bin doesn't exist.last_index must point at the real last token, not at padding.convert_coreml.py (coremltools 9, PyTorch 2.x). Running Qwen3 on the Neural Engine correctly took three changes:
CPU_AND_NE. With one function per fixed length, the Neural Engine runs the model.layer_norm([x, -x]). Qwen3's residual stream reaches about 6,000, so x² overflows fp16 (max 65,504) inside a plain RMSNorm, and the output collapses to cosine 0.05. Concatenating x and -x gives a zero-mean input, so LayerNorm's variance equals mean(x²). The two forms are mathematically identical, and the Neural Engine's fused LayerNorm stays in range.We also built and tested an 8-bit-weight version but didn't ship it: 10.7 ms per query and half the size, but its minimum cosine dropped to 0.9986.
| File | |
|---|---|
papr-embed-code-1-0.6b.mlpackage |
the model: multifunction, fp16 |
tokenizer.json, tokenizer_config.json |
Qwen3 tokenizer, same as the base model |
examples/embed.py |
runnable example |
convert_coreml.py |
conversion script |
results/coreml_parity.json |
latency and parity measurements |
Used in Papr for local-first code search. Apache-2.0, same as the base model.