🇰🇷 한국어 카드: MODEL_CARD.ko.md

EG2 Tiny Query Encoder A1

A 17.6 MB Korean/English query encoder that writes into the image embedding space of google/embeddinggemma-2. Keep your existing EmbeddingGemma 2 image index and search it without re-indexing.

The 17.6 MB is the text-encoder bundle only. A working search system also needs the image vectors from EmbeddingGemma 2 and, for the Korean chart result below, an OCR + BM25 keyword index. Neither is counted in that number.

Quick start

import sys
import numpy as np
from huggingface_hub import snapshot_download

repo = snapshot_download("ThakiCloud/eg2-tiny-query-a1")
sys.path.insert(0, repo)
from a1_query import A1QueryEncoder

enc = A1QueryEncoder.from_pretrained(repo)
q = enc.encode(["월별 매출 추이", "login screen with email and password"])
print(q.shape)  # (2, 768)

# Your existing index: (N, 768) L2-normalized EmbeddingGemma 2 image vectors.
image_vectors = np.load("image_vectors.npy")
top10 = np.argsort(-(q @ image_vectors.T), axis=1)[:, :10]
print(top10[:, :3])

Requirements: torch, transformers, safetensors, huggingface_hub, numpy. CPU is enough.

No index yet? example_search.py builds one with EmbeddingGemma 2 and then searches it with A1. Building the index needs pip install "sentence-transformers[image]>=6.1.0" (EmbeddingGemma 2's processor requires torchvision); searching does not.

python example_search.py build  --images ./images --index index.npz
python example_search.py search --index index.npz --query "2023년 지역별 인구 비율" --query "settings page with toggle switches"

Image vectors must come from google/embeddinggemma-2 encoding the image alone ({"image": PIL.Image} in sentence-transformers), L2-normalized. Vectors from another model, or from a quantized EmbeddingGemma 2 we have not tested with A1, are not supported.

What was measured

All numbers except the held-out publisher test at the end of this section are from development sets used during model selection. Everything is scored on the INT8 bundle in this repository. Intervals are 95% cluster-bootstrap intervals (2,000 resamples, seed 0) over the paired per-query difference. They capture query sampling, not training-seed variation.

English: app screens (RICO Screen2Words validation, 2,364 queries, one human summary per screen, 625 app clusters)

A1 − E0 in R@10: +0.003 [−0.009, +0.015]. This met the pre-registered non-inferiority bound (lower limit ≥ −0.02): adding Korean training did not cost English retrieval. A1 still trails the teacher by about 0.06 R@10.

Korean: charts (AI Hub 71706 development split, 3,000 chart-title queries over 22,959 charts, 211 source clusters)

• Combined with OCR keyword search, A1 improves R@10 by +0.023 [+0.019, +0.039] over OCR + BM25 alone. This is the use the model is meant for on Korean charts.
• On its own, A1 is worse than OCR + BM25 (−0.062 [−0.107, −0.052]) and worse than the teacher. Do not use it as the only retriever for Korean charts.
• R@1 does not improve under fusion (0.846 vs 0.851). The gain is in getting the right chart into the top 10.
• The fused gain is not larger than that of two earlier Korean-trained students with the same architecture (difference −0.0003). What A1 adds over them is the English result above.
• 2,993 of the 3,000 queries come from chart publishers that also appear in training. The result is about new charts from seen publishers.

Korean: charts from held-out publishers (AI Hub 71706, one-time test)

One group of 67 chart publishers was kept out of training and out of model selection. After A1 was frozen, it was scored once on their charts. The test rules were written down before the test was opened. The pool has 9,805 charts from 66 publishers, after removing near-duplicates of training and development images (teacher image cosine ≥ 0.95). The queries are 3,000 chart titles from 54 publishers, after removing titles that also appear in training text or in the development pool.

• Fused R@10 gain over OCR + BM25: +0.025 [+0.015, +0.033], with the interval resampled by publisher (54 clusters). This passed the pre-registered test (lower limit > 0). It is a combined-search improvement on charts from 71706 publishers the model never saw.
• A1 alone is clearly below OCR + BM25 here (−0.186 [−0.271, −0.088]) and below the teacher (−0.112 [−0.172, −0.035]). The gap is wider than on the development set.
• R@1 gets worse under fusion (0.705 vs 0.733). If your application shows only the top result, fusion with A1 can hurt.
• The queries are uneven across publishers: two publishers contribute 1,573 of the 3,000 queries. The publisher-level interval accounts for this, but the mean leans toward those two.
• These publishers are still from the same AI Hub chart collection. This is not a test on other kinds of Korean images.

Video and audio indexes: measured, not recommended (2026-10-20)

We checked whether this encoder can search BF16 EmbeddingGemma 2 video and audio vectors. The index and queries were the same as for the BF16 text tower; only the query encoder changed.

Every difference from BF16 has its 95% interval below zero. The encoder runs, but it was distilled for image vectors only and does not carry over to video or audio despite sharing the 768-d space. Use it for image indexes; for video and audio, encode queries with EmbeddingGemma 2 itself.

Not claimed

• General Korean image search (photos, documents, screenshots). Korean training text was chart titles and chart-QA text only.
• Search over real product or platform screens. The English screen result uses public Android app screens.
• Text-to-text retrieval. A1 was trained to match image vectors, not to replace the EmbeddingGemma 2 text tower for documents.
• Compatibility with quantized EmbeddingGemma 2 image vectors (not tested).
• Latency or memory on any device. Nothing here measures speed.
• Robustness across training seeds (one run).

Training

• Base: wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M text tower (MIT) @ a2a8c6eaa2549ad66eb7c31b85022bf58273a26c, plus a new 256→768 linear layer.
• Teacher: google/embeddinggemma-2 (Apache-2.0) @ 914f7f89142e33e77833254d9c9b90c3cef7303b, BF16. Query targets use its SearchQuery prompt; image targets are its image embeddings.
• Data: 203,865 texts. English: 78,715 RICO Screen2Words training captions. Korean: 125,150 texts from AI Hub 71706 chart titles (publisher-written) and AI Hub 71812 chart-QA text (model-generated in the source dataset). One group of chart publishers was held out of training entirely.
• Loss: 1 − cos(student, teacher query vector) + in-batch InfoNCE against teacher image vectors (τ = 0.05) + a preservation term on English rows that keeps A1 close to E0 (weight 1.0, fixed before training).
• Schedule: 3 epochs, batch 512, AdamW, lr 1e-4 (encoder) / 1e-3 (linear layer), 3% warmup, max 64 tokens, seed 0, one GPU.
• Quantization: after training, every 2-D weight with ≥ 4,096 elements is stored as symmetric row-wise INT8 (scale = row max / 127, fp16); other tensors fp16. All numbers above were scored after this step.

Files

model.safetensors sha256: c19e5de6be52923e879e562e9b27003bb67328ba3175e3c2e43e1b8e9e18429b

Data and licenses

• Model weights and code: MIT (see LICENSE; includes the TinyCLIP notice).
• No training data is redistributed here.
• RICO Screen2Words captions are CC BY 4.0 (Google Research). Screens come from the RICO dataset (UIUC / Google).
• This research used datasets from 'The Open AI Dataset Project (AI-Hub, S. Korea)'. All data information can be accessed through 'AI-Hub (www.aihub.or.kr)'. Datasets used: 71706 (chart image-text) and 71812 (visualization QA). Only the trained model is released here; the AI Hub data itself is not redistributed, as AI Hub terms require.
• The teacher, EmbeddingGemma 2, is Apache-2.0. A1 contains none of its weights.