Calorie Lens

A photo of a meal goes in; out come the ingredients on the plate, how many grams of each, and the nutrition of every ingredient and of the whole plate (kcal, protein, fat, carbs, fiber, sugars, saturated fat, sodium). It is two models fused into one network, trained on the Nutrition5k dishes, whose every ingredient was weighed on a scale, on plates weighed at a Finnish lunch line (FlavoriaFoodWeight1700) and on USDA FoodData Central.

Model

photo ─► backbone (DINOv2-B, or ConvNeXt-B) ─► 253 ingredient queries (cross-attention)
                                              ├─ is ingredient k on the plate?   (presence)
                                              └─ how many grams, if it is?       (grams)
         backbone ─► totals head ─► plate mass, kcal, macros
grams_k = P(k) x grams_k, rescaled to the plate mass, tilted toward the plate's kcal per gram
        ──►  253 neurons  =  stage 1's output        (the photo and its mirror image, averaged)
253 neurons ─► NutritionLayer (frozen, 253 x 8 per-gram table) ─► nutrition per ingredient + totals
food name ─► NutriText (own BPE tokenizer + 4-layer transformer, from scratch) ─► nutrients / 100 g

config.json records pretrained: false; model.load() rebuilds every network from config.json and loads model.safetensors, so inference never downloads a base model. The weights load with transformers versions before and after 5.18, which renamed DINOv2's attention layers.

Usage

Python

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("shalev396/calorie-lens")
sys.path.insert(0, path)
import model
predictor = model.load(path, device="cpu")            # or "cuda"
result = predictor.predict("meal.jpg")                # a PIL image, bytes or base64 work too
result = predictor.predict("meal.jpg", total_grams=420)   # plate weighed? the scale becomes exact
result, heatmaps = predictor.analyze("meal.jpg")      # + where each ingredient's query looked
predictor.calculate([("grilled salmon", 150), ("white rice", 200)])   # stage 2b food calculator

Space API (free): the Space exposes /predict (photo + optional total grams in, [result, seconds, device] out).

Inference Endpoint: deploy this repo from its page (Deploy -> Inference Endpoints). handler.py takes an image (Content-Type: image/jpeg, or JSON {"inputs": "<base64>", "parameters": {"total_grams": 350}}) and returns the result above; {"inputs": [["oatmeal", 250], ["banana", 120]]} runs the calculator.

Training

• Data: shalev396/calorie-lens-data:
  • Nutrition5k: one overhead RGB-D scan per dish where available and 5 frames from each of the 4 side-angle videos. Label bugs were fixed (Cafe 2 totals logged as 0, x1000 scale errors, empty names, duplicate table rows) and every dish was assigned a plate group (plates were scanned once per added item), so validation never shares a plate with training. Test = the official split.
  • FlavoriaFoodWeight1700: 856 lunch plates from an automated cafeteria line in Finland, photographed from above with every serving weighed (median 472 g, 138 plates over 600 g). They give the plate weight and, for 52 plain foods (rice, potatoes, tomato, ...), the grams of that ingredient; no calories. Split by lunch day: 555 train, 301 test. 15 % of the training samples.
• Recipe: see training/: AdamW with layer-wise learning-rate decay, warmup + cosine, bf16, EMA of the weights, 40 epochs with early stopping on validation; augmentation on the GPU (any rotation for overhead scans, which keeps the camera-height scale cue). Targets a sample doesn't have (Flavoria's calories) are masked out of the loss. The totals head is trained on both log error and absolute error.
• Selection: two backbones trained with the same recipe: ConvNeXt-B from ImageNet-22k weights (timm) and DINOv2-B. Each of them, and the shipped v1 model, is read with 1, 2 or 4 test-time views and gets its own readout fitted on validation; the lowest validation score ships. The test split is evaluated once, at the end.

Experiments

Validation picked DINOv2-B (14.92 vs 15.17; the gap is inside its 95 % CI of -1.78 to +1.45, and the test split agrees: 16.01 % vs 16.11 %). On the Finnish plates DINOv2-B is far better (25.5 % vs 51.5 %). Both new networks beat the v1 network, which reaches 17.46 % with the new readout (18.35 % with its old one): the readout is worth about 0.9 points, the new training about 1.5 more. DINOv2-B peaked at epoch 32 and then slowly overfit; ConvNeXt-B peaked at epoch 35. Training took 2.9 h on a Colab GPU (ConvNeXt-B 64 min, DINOv2-B 101 min, NutriText and evaluation the rest).

Evaluation

Numbers: overhead test scans (the official depth_test split, labels corrected), as MAE and percent MAE (MAE / mean of the truth, the Nutrition5k convention). metrics.json also has the side-angle views, test_clean (plates never seen in training), the original labels and the 95 % CI.

Headline: 16.01 % calorie error (40.9 kcal) on the 507 overhead test scans, 95 % CI 14.1-17.9 % (bootstrap over plate groups), down from 18.35 % for v1. The original labels give the same 16.01 % on these scans, so the number compares directly with published results. Plate mass: 12.6 % (25 g). Averaging all side-angle frames of a dish: 15.9 %. Plates never seen in training (test_clean): 16.3 %.

Large plates improved but are still underestimated: over 600 kcal (38 dishes) the error fell from 167 to 146 kcal, and from 78 to 64 kcal at 400-600 kcal. Under 100 kcal it is 12 kcal (v1: 11).

Another cafeteria. 301 Finnish plates from lunch days never trained on (weight only: the source has no calories):

v1 had never seen a plate over ~600 g and came out 207 g light on average. v2's totals head learned these plates: 16 % error and almost no bias. The itemised readout is worse there, because on these photos the model marks dozens of ingredients as present: Flavoria labels say which foods are on a plate but never which are absent, and the readout blends 15 % of that inflated gram sum into the plate mass.

What each part is worth. The totals head read on its own scores 15.7 %, close to the itemised readout's 16.0 % (in v1 the gap was 16.4 vs 18.35 %). Step by step: soft grams rescaled to the plate mass 16.25 %, tilted toward the kcal per gram 16.08 %, averaged over 2 views 16.01 %. The plain sum of the soft grams, without rescaling, is the worst (21.9 %).

Ingredients: presence mAP 0.69 (F1 0.75 at the display threshold 0.65); 10.1 g error per present ingredient. Each ingredient's query looks at its own part of the plate:

NutriText (stage 2b), on 633 USDA foods it never saw: kcal R² 0.89 vs 0.78 for a TF-IDF nearest-neighbour lookup, 31.2 vs 47.4 kcal per 100 g, and better on all eight nutrients. On the 253 Nutrition5k names (zero-shot): 52 vs 87 kcal per 100 g (median 19 vs 49).

Limitations

• The Nutrition5k photos come from two Google cafeterias (a fixed overhead camera height, 253 ingredients); the Finnish plates add a third cafeteria, for weight only. On other photos the mass scale is uncertain; entering the plate's weight (total_grams) removes that error completely.
• On photos that don't look like Nutrition5k the ingredient list is unreliable: it spreads over many likely dishes (a Finnish rice-and-ratatouille plate comes out as lasagna, mashed potatoes, pasta, rice, ...). The plate weight in direct_estimate holds up better there.
• Hidden oil, butter, dressings and sauces can only be inferred from how the food looks.
• Only kcal, fat, carbs and protein are Nutrition5k's own values; the other four nutrients come from a USDA food matched to each ingredient. NutriText values for names it never saw are estimates, and some are off (it gives grilled salmon 6.2 g of carbs per 100 g).
• Plates over ~600 kcal are still underestimated (see the chart above).
• Results on corrected labels are not directly comparable with published Nutrition5k numbers; the original-label results are in metrics.json.

Code licence: MIT. The image backbone was fine-tuned from DINOv2 weights (Apache-2.0, see NOTICE). Training data: Nutrition5k (CC BY 4.0, Thames et al., CVPR 2021), FlavoriaFoodWeight1700 (CC BY 4.0, Sarapisto et al., 2022) and USDA FoodData Central (public domain).