AlephBERT Hebrew Shopping Intent Classifier

סיווג כוונות בעברית

This is a small Hebrew text classifier. You give it a short message like "תוסיף חלב וביצים" ("add milk and eggs") and it tells you what the person wants, out of 17 possible intents (add an item, show the list, clear the list, and so on). It is a fine-tune of AlephBERT, a Hebrew BERT model, and I built it as a learning project.

I am not an ML researcher, and this is not a Hebrew NLP benchmark. It is a worked example of turning one narrow problem into a small fine-tuned model and checking it honestly. The full tutorial and code are in the GitHub repo: github.com/spivi/alephbert-intent-he.

Important limitation: the train and test data are synthetic paraphrases, not real chat traffic. The split avoids leakage by holding out source seeds before paraphrasing, but read the numbers as a controlled experiment, not as proof of real-world accuracy.

Try it

from transformers import pipeline

clf = pipeline("text-classification", model="spivi87/alephbert-intent-he", top_k=3)
clf("תקנה ביצים")
# [{'label': 'GROCERY_REQUEST', 'score': 0.90}, ...]

You get readable label names back (not LABEL_0). The mapping from numbers to names is saved inside config.json. If you would rather run without PyTorch, the GitHub repo has a one-command ONNX export.

There is also a live demo you can try in the browser, with no setup: spivi87/alephbert-intent-he-demo.

What it is for

It sorts short Hebrew messages from a shopping or grocery chat into one of 17 intents: add an item, show the list, clear the list, a recipe link, group admin actions, and so on (the full list is in the table further down).

It also works as a starting point if you want to fine-tune your own Hebrew classifier for a different topic. The weights are already comfortable with short, informal Hebrew that has typos, emoji, and a little English mixed in.

A practical tip: I use about 0.7 as a confidence heuristic in the demo. I did not calibrate the probabilities, so treat it as an operational rule of thumb, not a proven threshold. Below it, fall back to a larger model or treat the message as OTHER.

Results

Mean over 3 training runs (seeds 42, 43, 44), on a held-out test set of 374 messages (22 per intent). The published checkpoint is the seed 42 run.

• Accuracy: 0.773 ± 0.008
• Macro F1: 0.763 ± 0.008

Per-intent metrics (seed 42)

The full report and confusion matrix are in EVALUATION.md and confusion_matrix.png.

How it compares to a zero-shot LLM

On the same 374-message test set:

On this narrow synthetic grocery-intent task the fine-tune scored higher than my zero-shot GPT-4o-mini baseline on the same split. That does not mean it is a better general Hebrew model. It means that for a narrow, repeated task, a small task-specific model can be cheaper, private, and surprisingly effective. The GPT-4o-mini cost is an estimate for my zero-shot prompt and these short messages; OpenAI bills per token, not per message, so your cost will vary with prompt length, number of labels, and output format.

How it was trained (short version)

The training data is synthetic. I started from a handful of Hebrew example sentences per intent and asked an LLM to rewrite each one into many variations. After testing the first version I found gaps and fed the failures back into the data: "buy X" requests like "תקנה חלב" were misread, and short single-item requests like "תקנה ביצים" were confused with removing an item. I added hand-authored examples to cover the missing phrasings and retrained. No real user messages were used, so there is no private data. A sample is in the GitHub repo, under data/sample.jsonl.

The train/test split happens at the seed level, before the rewriting step. For each intent, 2 seeds are held out completely, and the test set only contains rewrites of those held-out seeds. Splitting after rewriting instead would let rewrites of the same sentence land on both sides, so the model has effectively seen the test questions and the accuracy looks better than it is.

Training settings

The full step-by-step instructions are in the GitHub repo.

Labels

Good to know before you rely on it

• It was trained only on shopping and grocery messages. For another topic, treat it as a base to fine-tune, not a finished classifier.
• BUG_REPORT and OTHER are the weakest classes (F1 around 0.32 to 0.64). Read a low-confidence prediction as "not sure" rather than a firm label.
• It expects Hebrew, in short messages of up to 128 tokens. A little Hebrew and English mixing is fine.
• The training data is synthetic, so unusual or very noisy phrasing may be classified less reliably than the test examples.

Credit and license

This model is only a fine-tune. The real work, teaching a model the Hebrew language in the first place, was done by the AlephBERT team at the ONLP Lab, Bar-Ilan University: Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. Thank you to them. Their code and models are at github.com/OnlpLab/AlephBERT. If AlephBERT is useful in your own work, please cite their paper.

Base model: onlplab/alephbert-base (Apache 2.0). This fine-tune is released under Apache 2.0 as well.

@inproceedings{seker-etal-2022-alephbert,
    title     = "{A}leph{BERT}: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level",
    author    = "Seker, Amit and Bandel, Elron and Bareket, Dan and Brusilovsky, Idan and Greenfeld, Refael and Tsarfaty, Reut",
    booktitle = "Proceedings of the 60th Annual Meeting of the ACL",
    year      = "2022",
    address   = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url       = "https://aclanthology.org/2022.acl-long.4",
    pages     = "46--56",
}