Evora on-device photo model (Qwen3-VL-2B, v2)

Recognises the meal in a photo for the Evora apps (iPhone and Android), on the phone itself. It answers Evora's meal.identify.image task: a short title in the user's language and the plate's components, each with an English food-database query, an estimated weight in grams and a confidence.

The model does not estimate calories or nutrients. The Evora apps send the recognised component queries and weights to the Evora server, which looks up the nutrition in USDA FoodData Central and Open Food Facts. The photo never leaves the phone.

Files

Prompt format

The model was trained on exactly one prompt shape; use it verbatim.

• System: You are Evora's on-device model. Task: meal.identify.image. Language: de (German). Reply with the JSON object only. (en (English) for English. Any other language is best asked in English.)
• User: the photo, directly followed by Identify the meal in this image. — or, with a hint from the user, Identify the meal. Hint: <hint>.
• Assistant: one compact JSON object.

As raw text (Qwen3-VL chat template; llama.cpp's mtmd replaces <__media__> with the image and adds the vision start/end tokens itself):

<|im_start|>system
You are Evora's on-device model. Task: meal.identify.image. Language: en (English). Reply with the JSON object only.<|im_end|>
<|im_start|>user
<__media__>Identify the meal in this image.<|im_end|>
<|im_start|>assistant

Photos were at most 640×640 pixels in training (about 400 image tokens); the apps downscale to that budget. Decode greedily (temperature 0), ideally under a JSON grammar.

Samples

Both plates are from Nutrition5k's held-out split, run through the published Q8_0 file with the same llama.cpp code the apps use (llama.cpp b11200, greedy, JSON grammar). The "measured" column is Nutrition5k's own ingredient list and scale weight for the plate.

German

System

You are Evora's on-device model. Task: meal.identify.image. Language: de (German). Reply with the JSON object only.

User — the photo above, then

Identify the meal in this image.

Assistant

{"title":"Apfel, Erdbeeren, Mandeln und Orange","components":[{"name":"Apfel","query":"apple raw","estimated_weight_grams":147,"confidence":0.96},{"name":"Erdbeeren","query":"strawberries raw","estimated_weight_grams":78,"confidence":0.97},{"name":"Mandeln","query":"almonds raw","estimated_weight_grams":42,"confidence":0.95},{"name":"Orange","query":"orange raw","estimated_weight_grams":48,"confidence":0.94}]}

English

System

You are Evora's on-device model. Task: meal.identify.image. Language: en (English). Reply with the JSON object only.

User — the photo above, then

Identify the meal in this image.

Assistant

{"title":"Roast beef with mixed vegetables and pearl barley","components":[{"name":"Roast beef","query":"roast beef","estimated_weight_grams":112,"confidence":0.84},{"name":"Mixed green salad","query":"mixed green salad","estimated_weight_grams":73,"confidence":0.8},{"name":"Broccoli","query":"broccoli cooked","estimated_weight_grams":74,"confidence":0.94},{"name":"Pearl barley","query":"pearl barley cooked","estimated_weight_grams":78,"confidence":0.82}]}

Typical of the model: the main components right, a lookalike now and then (pears as apple), and hidden fats (oil, dressing) missed, which is most of the gap in total weight.

Running it

With llama.cpp's server (OpenAI-compatible API):

llama-server -m evora-qwen3-vl-2b-photos-v2.Q8_0.gguf \
  --mmproj mmproj-evora-qwen3-vl-2b-photos-v2.f16.gguf --jinja --temp 0

Send the system prompt above, and a user message with the image followed by the text Identify the meal in this image.; ask for "response_format": {"type": "json_object"}.

Evaluation

30 held-out Nutrition5k plates (15 German, 15 English prompts), merged model before quantization:

Training

• Base: Qwen/Qwen3-VL-2B-Instruct, LoRA (r 32, alpha 64) on the language model, merged, converted and quantized with llama.cpp.
• Data: about 3,900 overhead photos of cafeteria plates from Nutrition5k (Google, CC BY 4.0), prompted in German and English. Titles, components and queries were written by gpt-5.4 answering Evora's production prompt; the gram values were then rescaled so each plate's components add up to the weight Nutrition5k measured on a scale. v1 learned from the teacher alone; v2 continued from v1 on the rescaled weights.
• Trained in AI Studio on one AMD RX 6900 XT.

Limitations

• Nutrition5k is one cafeteria's food, photographed from above on a plain background. Home cooking, restaurant plates and photos at an angle are harder, and regional dishes may be named as their nearest cafeteria equivalent.
• Weights are estimates from a single photo; sauces, oil and anything hidden under other food are routinely underestimated.
• Not a medical device and not for clinical nutrition decisions.

Licence

Apache-2.0, as the base model. Training photos: Nutrition5k, CC BY 4.0 (Thames et al., Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food, CVPR 2021).