Evora on-device photo model (Qwen3-VL-2B, v3)

Recognises the meal in a photo for the Evora apps (iPhone and Android), on the phone itself. It answers Evora's meal.identify.image task: a short title in the user's language and the plate's components, each with an English food-database query, an estimated weight in grams and a confidence.

The model does not estimate calories or nutrients. The Evora apps send the recognised component queries and weights to the Evora server, which looks up the nutrition in USDA FoodData Central and Open Food Facts. The photo never leaves the phone.

Files

Prompt format

The model was trained on exactly one prompt shape; use it verbatim.

• System: You are Evora's on-device model. Task: meal.identify.image. Language: de (German). Reply with the JSON object only. (en (English) for English. Any other language is best asked in English.)
• User: the photo, directly followed by Identify the meal in this image. — or, with a hint from the user, Identify the meal. Hint: <hint>.
• Assistant: one compact JSON object.

As raw text (Qwen3-VL chat template; llama.cpp's mtmd replaces <__media__> with the image and adds the vision start/end tokens itself):

<|im_start|>system
You are Evora's on-device model. Task: meal.identify.image. Language: en (English). Reply with the JSON object only.<|im_end|>
<|im_start|>user
<__media__>Identify the meal in this image.<|im_end|>
<|im_start|>assistant

Photos were at most 640×640 pixels in training (about 400 image tokens); the apps downscale to that budget. Decode greedily (temperature 0), ideally under a JSON grammar.

Samples

Held-out photos (never trained on), run through the published Q8_0 file with the same llama.cpp code the apps use (llama.cpp b11200, greedy, JSON grammar). For the Nutrition5k plates, "measured" is Nutrition5k's own ingredient list and scale weight.

German — a Nutrition5k plate

System

You are Evora's on-device model. Task: meal.identify.image. Language: de (German). Reply with the JSON object only.

User — the photo above, then

Identify the meal in this image.

Assistant

{"title":"Gemüse- und Fruchtsalat mit Mandeln und Orange","components":[{"name":"Papaya","query":"papaya raw","estimated_weight_grams":120,"confidence":0.94},{"name":"Amandeln","query":"almonds raw","estimated_weight_grams":30,"confidence":0.96},{"name":"Orangenscheiben","query":"orange segments","estimated_weight_grams":45,"confidence":0.98},{"name":"Erdbeeren","query":"strawberries raw","estimated_weight_grams":40,"confidence":0.97}]}

Papaya where the scale says pears, and "Amandeln" for Mandeln: the kind of slip the model still makes. The weight is on the low side, as it is for most small cafeteria plates.

English — a Nutrition5k plate

System

You are Evora's on-device model. Task: meal.identify.image. Language: en (English). Reply with the JSON object only.

User — the photo above, then

Identify the meal in this image.

Assistant

{"title":"Beef with mixed vegetables and pearl barley","components":[{"name":"Beef","query":"beef steak cooked","estimated_weight_grams":120,"confidence":0.82},{"name":"Mixed vegetables","query":"mixed vegetables cooked","estimated_weight_grams":140,"confidence":0.88},{"name":"Pearl barley","query":"pearl barley cooked","estimated_weight_grams":100,"confidence":0.84}]}

English — a full-size plate

Photo: FitTasteTic, CC BY-SA 2.0, via Wikimedia Commons.

System

You are Evora's on-device model. Task: meal.identify.image. Language: en (English). Reply with the JSON object only.

User — the photo above, then

Identify the meal in this image.

Assistant

{"title":"Spaghetti with bolognese sauce","components":[{"name":"Spaghetti","query":"spaghetti cooked","estimated_weight_grams":180,"confidence":0.96},{"name":"Bolognese sauce","query":"bolognese sauce","estimated_weight_grams":140,"confidence":0.92},{"name":"Parmesan cheese","query":"parmesan cheese","estimated_weight_grams":10,"confidence":0.94}]}

330 g for a dinner plate of spaghetti bolognese (gpt-5.4: 412 g; v2 at Q4_K_M: 286 g). There is no scale for photos like this one; the evaluation below compares them with the teacher.

Running it

With llama.cpp's server (OpenAI-compatible API):

llama-server -m evora-qwen3-vl-2b-photos-v3.Q8_0.gguf \
  --mmproj mmproj-evora-qwen3-vl-2b-photos-v3.f16.gguf --jinja --temp 0

Send the system prompt above, and a user message with the image followed by the text Identify the meal in this image.; ask for "response_format": {"type": "json_object"}.

Evaluation

60 held-out photos, run through the Q4_K_M files of v2 and v3 with the apps' llama.cpp code (greedy, JSON grammar): 30 Nutrition5k plates, whose weight was measured on a scale, and 30 full-size plates (restaurant and home cooking, from Open Images and Wikimedia Commons), compared with gpt-5.4's estimate for lack of a scale.

Unquantised, v2 was 25% off the scale on the same Nutrition5k plates; most of the gap to the Q4_K_M figure is quantisation, which is why phones with 8 GB get the Q8_0 file.

Training

• Base: Qwen/Qwen3-VL-2B-Instruct, LoRA (r 32, alpha 64) on the language model, merged, converted and quantized with llama.cpp.
• v1–v2: about 3,900 overhead photos of cafeteria plates from Nutrition5k (Google, CC BY 4.0), prompted in German and English. Titles, components and queries were written by gpt-5.4 answering Evora's production prompt; the gram values were then rescaled so each plate's components add up to the weight Nutrition5k measured on a scale.
• v3: Nutrition5k's plates are small (median 171 g) and shot from above, and v2 carried that over to every plate: a dinner plate of spaghetti came out at 250–300 g. v3 continued from v2 on 5,139 rows: the Nutrition5k rows plus 1,687 full-size meals (of 1,864 photos; the rest held out) from Open Images V7 and Wikimedia Commons, median 380 g, labelled by gpt-5.4 with the photo in front of it. 60 steps.
• Trained in AI Studio; v3 on a rented RTX PRO 4500 (RunPod, Secure Cloud).

Limitations

• Weights are estimates from a single photo; sauces, oil and anything hidden under other food are routinely underestimated.
• Small cafeteria plates still come out on the low side, and regional dishes may be named as their nearest common equivalent.
• Not a medical device and not for clinical nutrition decisions.

Licence

Apache-2.0, as the base model. Training photos: Nutrition5k, CC BY 4.0 (Thames et al., Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food, CVPR 2021); Open Images V7 (700 photos, CC BY 2.0); Wikimedia Commons (1,164 photos: CC BY, CC BY-SA, CC0 and public domain). The photos are not part of this repository; training-photos.csv credits each full-size one with its source, author and licence.