EmbeddingGemma 2 · Multimodal Typed Decision

A typed-decision model fine-tuned from EmbeddingGemma 2 (Google DeepMind). Given a state and a set of typed questions, it returns calibrated probabilities for every option of every question in a single call, without generating text.

• State: structured or free text, optionally with images or video.
• Question types: choice (pick one of named options), noul (yes/no) and score (ordinal levels).

Each answer is a probability over options, so it works as a typed classifier, router, tagger, policy check or tool selector.

This model follows the typed-decision interface of mrm8488/embeddinggemma-2-ft-typed-decision. It adds:

• native image and video input, because the EmbeddingGemma 2 vision encoder is retained;
• a single-pass joint encoder for fast decisions;
• multitask training on public train splits from many decision tasks;
• an optional box head that locates the target of a question in an image.

Highlights

Overall typed accuracy is statistically indistinguishable from the previous release (paired-record bootstrap 95% CI of the difference: −1.75 to +0.95 percentage points). The suite has ten significant improvements and one significant regression (ACOS, −0.93 points) under the pre-registered, Holm-adjusted comparison. Workflow exact-match accuracy remains low.

The optional box head retains its existing encoder, adapter and heads. RefCOCO testB Acc@0.5 is 85.3%; ImageJevBench Box Acc@0.5 is 78.7%. See the box section for caveats and calibration.

How it works

Two separately fine-tuned EmbeddingGemma 2 members answer decision questions. Exactly one member runs per request: the joint member reads state, questions and options together; text requests whose rendered state exceeds 2,048 tokens use the long-context member, which encodes state, question and options separately. Images and videos use the joint member. The threshold was selected on development data.

The joint head uses a two-layer cross-attention refiner over option spans and state tokens. Custom yes/no criteria are included in the option text. Its text-only long-input mode uses overlapping state windows and max-pools logits when a joint pass cannot fit. The long-context member windows states beyond 8,192 tokens when long_text_mode="chunked" is enabled. The vision tower and token embeddings remain frozen. Box questions use a third, dedicated encoder so decision-weight updates do not change their predictions.

Usage

Inference uses the bundled gemma_decision package. It is not loaded by AutoModel.

pip install "torch>=2.11" "transformers>=5.19" safetensors pillow av huggingface_hub

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("Danny-Dasilva/embeddinggemma-2-multimodal-typed-decision")
sys.path.insert(0, path)
from gemma_decision.typed import load_typed_model

model = load_typed_model(path, device="cuda", local_only=False,
                         long_text_mode="chunked", context_from_question=True)

response = model.decide({
    "state": {"title": "Fixing a kitchen sink trap leak",
              "transcript": "Place a bucket under the P-trap, loosen the slip nuts with pliers ..."},
    "questions": {
        "category": {"type": "choice", "instructions": "Primary category of this video?",
                     "criteria": {"plumbing": "Pipes, drains, faucets or water flow.",
                                  "electrical": "Wiring, outlets, breakers or lighting.",
                                  "woodworking": "Cutting wood and joinery."}},
        "difficulty": {"type": "score", "instructions": "Skill level required?",
                       "criteria": ["Beginner", "Intermediate", "Advanced"]},
        "hazardous": {"type": "noul", "instructions": "Does it involve hazardous activity?"},
    },
})
print(response["answers"]["category"])   # {"choice": ..., "probabilities": {...}, ...}

# Images and video: pass local file paths; they enter the same joint encoder pass.
model.decide({"state": "Classify the attached video.", "videos": ["clip.mp4"],
              "questions": {"activity": {"type": "choice", "instructions": "Which activity?",
                                         "criteria": {"surfing": "surfing", "skiing": "skiing"}}}})

Command line:

PYTHONPATH=$path python -m gemma_decision.cli predict --checkpoint $path --data inputs.jsonl \
  --device cuda --context-from-question --long-text-mode chunked --output predictions.jsonl

• CUDA dtype: bfloat16 is used on CUDA. Do not use float16.
• Frozen media route: media_route="frozen" sends images and video to a frozen-encoder media head instead. It downloads the pinned google/embeddinggemma-2 base.
• Context adapter: context_from_question moves the input text of a single empty-state choice prompt ("Header:\n<text>") into the state. The model was trained with it.

Box (object localisation)

An optional add-on answers Jev-style box questions: where in an image is the thing the question describes. It uses a 4.9M-parameter main head, a crop-refinement head and a rank-16 language-model adapter (box/). The adapter is merged into a copy of the dedicated box language model (box/backbone/); its encoder weights are preserved from the box training base. Questions of other types are delegated to the updated decision model with its full routing. Loading the optional box wrapper uses additional encoder memory; decision-only loading does not load it.

• A box question names its target in text ("the coffee mug in `photo`"), or with a reference picture ("the object that matches this picture: <image>").
• Mode decoding groups image tokens whose boxes overlap the most likely box (IoU > 0.3). The answer is one box: four edges as 0–1 fractions of the searched image, measured from its top-left corner.
• confidence is P(present) × P(IoU ≥ 0.5 | present), the probability that the box is right. presence (is the target there at all) and count (none / one / several) are returned as extra fields.
• A box question gets its own encoder pass over the state, the record images and the question. The head reads five hidden-state layers of that pass at the image and question tokens. Several box questions on one record share one batched pass.

Request and response

{"state": {"photo": "<image 0>"},
 "images": ["photo.jpg"],
 "questions": {
   "racket": {"type": "box", "instructions": "the tennis racket in `photo`"},
   "outdoors": {"type": "noul", "instructions": "Was this photo taken outdoors?"}}}

Illustrative response shape:

{"racket": {"type": "box",
            "box": {"top": 0.626, "bottom": 0.749, "left": 0.630, "right": 0.756},
            "confidence": 0.963, "presence": 1.000, "count": "one"},
 "outdoors": {"type": "noul", "probabilities": {"false": ..., "true": ...}, ...}}

Box question fields:

gemma_decision.box.jev_request(request, asset_paths) converts a Jev request that uses asset references into this record shape.

Usage

from gemma_decision.box import load_box_model

# Same options as load_typed_model; the head is read from <path>/box (box_head=... for another directory).
model = load_box_model(path, device="cuda", local_only=False,
                       long_text_mode="chunked", context_from_question=True)
response = model.decide({
    "state": {"photo": "<image 0>"},
    "images": ["photo.jpg"],
    "questions": {
        "racket": {"type": "box", "instructions": "the tennis racket in `photo`"},
        "match": {"type": "box", "instructions": "the object that matches this picture: <image>",
                  "images": ["reference.jpg"]},
        "outdoors": {"type": "noul", "instructions": "Was this photo taken outdoors?"},
    },
})
print(response["answers"]["racket"])

Records without box questions go straight to the decision model, so load_box_model(path) can replace load_typed_model(path). Command line: add --box-head (it reads $path/box; give a directory to use another head):

PYTHONPATH=$path python -m gemma_decision.cli predict --checkpoint $path --box-head --data inputs.jsonl \
  --device cuda --context-from-question --long-text-mode chunked --output predictions.jsonl

Results

Metrics below use the saved evaluation predictions with automatic refinement for boxes smaller than 10% of the image. Architecture, decoding and refinement settings were selected on development data. Acc@0.5 and mIoU score present targets with boxes. AUROC compares raw presence on present and absent targets. Calibration is 100 × (1 − ECE / 0.5), computed on present boxed targets only; it does not measure confidence on absent targets. No post-hoc confidence map is applied.

Referring expressions (all expressions of each split):

Categories, picture queries, presence and parts (n = boxed present + absent targets):

For the parts set, faces reach 94.4% Acc@0.5 and hands 59.7%. Hand results vary by dataset (53.3% on COCO-WholeBody and 90.3% on Open Images). The count set has 86.6% count accuracy; single-instance box accuracy is 90.6%. Attribute-level crop negatives contain some ambiguous labels.

Calibration. Referring-expression scores range from 88.2 to 94.8. LVIS scores 81.1, attribute crops 75.4, and unfamiliar targets 80.0. On absent targets, the share with confidence above 0.5 is 1.5% for LVIS, 1.8% for COCO categories, and 13.8% for attribute crops.

ImageJevBench Box

ImageJevBench Box track: 300 items, 200 COCO photos, 50 rendered documents and 50 rendered UI screens (items and method, MIT). Requests use the state {"photo": <image>}. The Jev column is its published Jev 1.14 result (2026-10-14); Jev was not re-run here.

Documents and UI remain harder: Acc@0.5 is 52.0% and 46.0%, respectively. ImageJevBench describes its photos as COCO val2017, but 29 are COCO train2017 images. None of the 29 is in the final training data; nine were present during an earlier training phase before removal, so these scores may be optimistic. The hosted and local latency numbers measure different systems.

Speed

RTX 5090, batch 1, bfloat16, warm, one decide() call per question, image decode and preprocessing included. Automatic refinement is included in these measurements.

Caveats

• One box per question. With several instances, the box covers one instance or can span several.
• Documents, UI and hands. Performance depends on target size and image source; document and UI accuracy remains below natural-image accuracy.
• Seen images. The decision encoder saw A-OKVQA train images without box supervision. These include RefCOCO-family validation/test images. COCO val2017 sets do not have that overlap.
• Relational phrasing. Synthetic relational training queries share templates with ImageJevBench referring items, on different images.
• Confidence and presence. Confidence estimates whether the box is correct; presence estimates whether the target exists. Calibration is weaker on some target families.
• Evaluation reuse. These sets have been inspected in multiple development rounds. This inference selection was the second held-out look in its evaluation protocol; the candidate family was designed after the first look. Selection within that family used development data only.
• Batching. Tables use batched evaluation. Padding and bfloat16 can change individual boxes in single-record calls.
• Film-frame text. Presence AUROC on the held-out film demo is 0.942, with precision 0.63 and recall 0.88 at presence ≥ 0.5. These are presence scores, not text-box localisation scores.

Box training data

A box-specific language-model adapter, the main head and the crop-refinement head were trained. The decision weights, vision tower and token embeddings were left unchanged. Training and development images, seeds and videos are separated; referring-expression evaluation image IDs are excluded from the box training data.

• RefCOCO, RefCOCO+ and RefCOCOg: train expressions and same-category negative crops; refer code is Apache-2.0. The jxu124 dataset copies do not state a dataset licence; underlying COCO image terms apply.
• COCO and LVIS: categories, relations, count and part targets, CC BY 4.0 annotations; images retain their individual Flickr licences.
• Visual Genome: hand boxes, CC BY 4.0 annotations, without COCO-overlap images.
• Open Images: hand and unfamiliar-category targets, CC BY 4.0 annotations; selected images are CC BY 2.0.
• COCO-Text: word and text regions, CC BY 4.0 annotations; COCO images retain their own licences.
• DocLayNet: page regions and text, CDLA-Permissive-1.0.
• Generated documents and UI: own synthetic content, plus text overlays rendered with system fonts under their respective font licences.
• Video frames: Big Buck Bunny and Elephants Dream (Blender Foundation, CC BY 3.0 / CC BY 3.0 US); Wikimedia Commons aerial videos under CC BY, CC0 or public domain terms. Train and evaluation videos are disjoint.
• Evaluation only: COCO-WholeBody annotations (CC BY-NC 4.0), ImageJevBench (MIT; document/UI images CC BY 4.0), and Sintel (Blender Foundation, CC BY 3.0).

Full benchmarks

Typed decisions

LocalLLaMA/typed-decisions test: 400 records / 2,000 decisions against published gold. The reference was measured locally in fp32.

Accuracy difference versus the reference: −0.50 percentage points, 95% paired-record bootstrap CI [−2.30, +1.35], exact McNemar p = 0.613. Overall accuracy is statistically indistinguishable; probability losses are lower. Yes/no accuracy fell from 85.00% to 82.67% against the previous published model; its Holm-adjusted McNemar p is 0.101.

Decision Index 0.2.1 (native suite)

Official Decision Index scoring on the available corpus: 42 scored benchmarks, 150,246 completed requests. HLE and interactive environments are excluded, so this is not an official full-suite aggregate. The chance-corrected index is 24.45. Clef and Laya columns are historical published scores, retained for context; neither was rerun. The teachers used for distillation were themselves trained for this suite, which can inflate gains.

Workflow evals

Public WorkflowEvals, consensus labels. This is the fourth look at these replay sets during the experiment series. The Jev reference is the frozen Jev 1.13.0 replay, not a new hosted measurement.

O*NET task-typing performance is 0.6814 (previous published model: 0.6831). Image/video decision quality was not remeasured for these weights; the older release's numbers are omitted. The box section reports the preserved box model's evaluations.

Decision speed

RTX 5090, bfloat16, full request wall time including preprocessing and all encoder passes; the machine was shared with other jobs. Results depend on input and schema size.

Per-benchmark latency is in the table and results/evaluation.json. Box latency and its distinct evaluation conditions are reported in the box section.

Training

Base. The google/embeddinggemma-2 text language-model layers were fine-tuned at revision 914f7f8. The vision encoder and token embeddings stayed frozen. Each member has a ~1.1M-parameter schema-conditioned head. The joint member also has a cross-attention refiner, distilled from an ensemble of joint and separately-encoded teachers on the training data. Per-type temperatures were calibrated on typed-decisions validation.

Data. Public source train partitions were used for training. Development subsets and synthetic held-out development sets selected epochs, rendering and routing. Test scores were not used to retune this candidate after selection, but the evaluation corpus has been reused across earlier development rounds:

• Typed decisions: LocalLLaMA/typed-decisions (Apache-2.0).
• Reasoning / NLI:
  • ARC
  • MMLU auxiliary_train
  • WinoGrande
  • HellaSwag
  • ANLI (CC BY-NC 4.0)
  • ContractNLI (CC BY 4.0)
  • NLI4CT
  • VAST
  • HoVer
  • RAGTruth (MIT)
• Intents / retrieval / commerce:
  • BANKING77
  • CLINC150
  • Amazon ESCI (Apache-2.0)
• Tools and function calling:
  • ToolRet-Training (no license stated)
  • When2Call (CC BY 4.0)
  • xLAM function-calling 60k (CC BY 4.0)
  • ToolACE (Apache-2.0)
  • glaive-function-calling-v2 (Apache-2.0)
  • API-Bank (MIT)
• Forecasting / humor:
  • ForecastBench questions resolved before the evaluation window (CC BY-SA 4.0)
  • SemEval-2020 Task 7 subtask-2 train
• Images / video:
  • A-OKVQA (Apache-2.0)
  • CIFAR-10
  • UCF101 official split-1 training groups

Data hygiene. Training rows were removed if they matched any evaluation request in any of these ways:

• they share a normalized 13-gram with it (prompt-template lines stripped);
• they reach bag-of-words Jaccard ≥ 0.8 with it in the same family;
• they share an ANLI premise, or a family-specific item key (document, statement, post, query, response or claim).

Development splits are group-disjoint from training. Known residual overlap: the long-context member's training data shares 32 of 1,592 HotpotQA source groups with the HoVer evaluation; the claims themselves differ.

Continued training and distillation. The joint member continued for three epochs and the long-context member for two. This adds public typed-decision and procedural corpora, CUAD contract windows (CC BY 4.0), GoEmotions ordinal labels (Apache-2.0), catch-all and out-of-scope schemas, generated workflow states, counterfactuals and schema paraphrases. The public corpora include tasksource-jev-typed-decisions (only rows marked commercial with permissive per-row licences) and procedural-typed-decisions (Apache-2.0). Dataset source partitions and group boundaries were retained.

Workflow data covers billing, disputes, security, agent traces and operational policies, with 17,402 training records, 835 validation records and 2,116 held-out development records. The last split has disjoint seed namespaces and subgenres. Rule-derived labels are blended 75% exact / 25% teacher where available. Other soft labels average the available Clef-flash and d3 predictions; Qwen3.5-9B supplies generation and selected intent labels. Earlier self-distillation targets and training data are retained. Teacher readouts were independently implemented and behaviorally checked, not verified bit for bit against their official runtimes.

Long-context training used four-record gradient microbatches after a memory failure, preserving the question-weighted optimizer objective and batch size. Dropout draws and packing can differ. Synthetic held-out labels come from the same teachers as training, so their gains measure distillation transfer rather than independent human gold.

Limitations

• Evaluation reuse and teacher contamination. Typed tests and the native suite have been inspected repeatedly across the experiment series. This candidate's selection was frozen before its test look, but its family was designed with knowledge of prior results. Clef-flash and d3 were trained for Decision Index tasks. Scores are not clean held-out estimates.
• Workflow reasoning. Exact-action accuracy remains low on invoices and customer service; the model is far behind the frozen hosted workflow reference. More synthetic examples alone have not closed that gap.
• Knowledge-heavy tasks. The small text encoder remains weak on knowledge and reasoning benchmarks. Not every task improved; see the complete table.
• SGD. The benchmark adapter includes future dialogue turns in some rows, creating conflicting labels.
• Language and runtime. Most data is English. Inference needs the bundled runtime.
• Box. Documents, UI and hands remain difficult; see the box caveats and evaluation reuse.
• Licensing. Source licences vary, including non-commercial ANLI, research-only UCF101 and ToolRet-Training with no stated licence. Source data is not redistributed here.

License and attribution

• License. This model is released under the Apache License 2.0. It is a fine-tuned derivative of EmbeddingGemma 2 by Google DeepMind, also Apache 2.0.
• Upstream projects. The rendering and scoring head follow the typed-decision design of mrm8488/embeddinggemma-2-ft-typed-decision; no reference weights were used. Evaluation uses the Decision Index and WorkflowEvals kits.
• Box data. RefCOCO, RefCOCO+ and RefCOCOg, COCO, LVIS, Visual Genome, Open Images, COCO-Text and DocLayNet; see Box training data.

Citation

@misc{embeddinggemma2_multimodal_typed_decision_2026,
  title  = {EmbeddingGemma 2 Multimodal Typed Decision},
  author = {Dasilva, Danny},
  year   = {2026},
  url    = {https://huggingface.co/Danny-Dasilva/embeddinggemma-2-multimodal-typed-decision},
  note   = {Fine-tuned from google/embeddinggemma-2}
}

@misc{embeddinggemma2_2026,
  title  = {EmbeddingGemma 2},
  author = {{Google DeepMind}},
  year   = {2026},
  url    = {https://huggingface.co/google/embeddinggemma-2}
}

@article{schechtervera2025embeddinggemma,
  title   = {EmbeddingGemma: Powerful and Lightweight Text Representations},
  author  = {Schechter Vera, Henrique and Dua, Sahil and Zhang, Biao and others},
  journal = {arXiv preprint arXiv:2509.20354},
  year    = {2025}
}