CAMeLBERT-mix — Apple Core AI, Arabic

A Core AI (.aimodel) build of CAMeL-Lab/bert-base-arabic-camelbert-mix for on-device inference on Apple Silicon.

Upstream publishes pytorch_model.bin, flax_model.msgpack and tf_model.h5 — and no MLX, no ONNX and no Core AI. This is the Core AI build.

Read this before using the vectors

The pooling is ours, not upstream's. CAMeLBERT-mix is a masked-language model; it publishes no embedding pipeline. This bundle adds mean pooling over the attention mask followed by L2 normalisation, which is the conventional choice — but it is a choice, and it is not something upstream optimised for.

The consequence is visible in the numbers below and should not be hidden: raw BERT embeddings are anisotropic. On the single unrelated pair measured below the score was 0.65, which is high for two sentences that share nothing. Corpus-wide anisotropy was not evaluated — one pair is one pair.

If you need well-separated Arabic sentence vectors, a sentence-transformers model is the better tool. If you need this model's representations on-device, unchanged and verified, this is it.

Verified equivalence

Measured against the original PyTorch encoder on six fixed inputs at the exported shape:

Worst across all six: 1.19e-07 — float32 rounding, not a modelling difference.

Arabic behaviour, stated honestly

On these three pairs the related ones rank 0.21 to 0.26 above the unrelated one — real, modest, and measured on three pairs rather than a corpus. Cross-lingual: "How do I reset my password?" sits at 0.7149 from its Arabic counterpart.

What is in here

Use

import asyncio, numpy as np
from coreai.runtime import AIModel, NDArray, SpecializationOptions
from transformers import AutoTokenizer

SEQUENCE_LENGTH = 512        # fixed shape: pad to it, then read the vector back

async def embed(text: str):
    tok = AutoTokenizer.from_pretrained(".")
    model = await AIModel.load("camelbert-arabic.aimodel", SpecializationOptions.default())
    fn = model.load_function("main")
    enc = tok([text], padding="max_length", truncation=True, max_length=SEQUENCE_LENGTH,
              return_tensors="np")
    out = await fn({"input_ids": NDArray(enc["input_ids"].astype("int32")),
                    "attention_mask": NDArray(enc["attention_mask"].astype("int32"))})
    return np.asarray(list(out.values())[0].numpy())

print(asyncio.run(embed("كيف أستعيد كلمة المرور؟")).shape)   # (1, 768)

How it was converted

torch.export at a fixed (1, 512) shape, run_decompositions, then coreai-torch 0.4.3. The graph takes int32 inputs, the only integer type the runtime accepts. The MLM head (cls.predictions.*) is not part of the graph — only the encoder; those keys load as UNEXPECTED when the checkpoint is read as an encoder, which is expected.

Limits

• Fixed at 512 tokens; pad rather than passing a shorter sequence.
• Only the encoder. Masked-token prediction is not in this graph.
• Six Arabic sentences and one English were measured. Arabic dialect variation, long documents and the 512-token boundary were not.
• No speed or memory benchmark. Equivalence only.

Licence and attribution

Apache-2.0, as declared by upstream. A derivative work of CAMeL-Lab/bert-base-arabic-camelbert-mix; credit for the model is theirs.