🇰🇷 한국어 카드: MODEL_CARD.ko.md

EG2 Mixed-Bit 550MB

EmbeddingGemma 2 at 550 MB, with 4, 6 or 8 bits per weight matrix chosen by measured sensitivity. On a sealed evaluation run once, text→image and text→video retrieval stayed within the pre-registered margin of BF16.

Status after the final evaluation (sealed set, opened once on 2026-10-16).

• Image and video retrieval: non-inferior to BF16 within the pre-registered margin (verdict IMAGE_VIDEO_GO).
• Audio: no sealed verdict. The sealed audio set failed its independence audit, so the sealed result stays INCONCLUSIVE. A later check on two development audio sets (2026-10-20, below) found no drop against BF16.
• Multimodal (text + image + audio + video together): NO_MULTIMODAL_GO, because audio could not be evaluated.
• Text-only retrieval: not measured. The text tower is quantized here, unlike ThakiCloud/eg2-hybrid-w4-media.
• Rebuild reproduction: CONTENT_REPRO across GPU types (the H100 build and a B200 rebuild decoded to identical tensors).
• Mixed-bit vs uniform at the same size: superiority not established (details below).

Bits per weight matrix, by tower (466 matrices):

Bit allocation. For each weight matrix we quantized that matrix alone (to 4 and to 6 bits, everything else BF16) and measured how much of the BF16 top-10 image-retrieval neighborhood survived, on a 256-query development image subset. Starting from all 4 bits, a greedy pass promotes whichever matrix buys the most retained neighborhood per added byte (4→6, 6→8) until the 550,000,000-byte budget is used. Assumptions, stated rather than tested: 8 bits loses nothing, the per-matrix losses add up, and the audio tower stays at 4 bits because the sensitivity data contains no audio.

Sealed evaluation

One evaluation job, opened once, measured every configuration with the same code, the same sentence-transformers stack, and the same queries, documents and answer keys. Each configuration encodes its own queries and builds its own index.

• Image (text → image): 2,113 queries over 2,121 images, 448 source-video clusters. Korean captions; AI Hub dataset 71866 (validation), split from the development images by source video and perceptual-hash component.
• Video (text → video): 1,252 queries over 1,722 video segments, 610 clusters (71866 source broadcast, DiDeMo uploader). 671 Korean-caption queries from 71866 and 581 English-description queries from DiDeMo (CC-BY videos). Segments with several correct answers are scored as a hit if rank 1 is any of them.
• Margin: δ = 10% of BF16 R@1 on this same sealed set: δ = 0.0321 (image), 0.0273 (video).
• Interval: per-query paired difference in R@1, cluster bootstrap (resample whole clusters with replacement), 2,000 resamples, percentile 95% interval, seed 0.
• Rule (fixed before scoring): non-inferior if the lower bound against BF16 is ≥ −δ, and the lower bound against the 872 MB reference is also ≥ −δ; size ≤ 654,000,000 bytes.
• Which checkpoints could receive the verdict: at most two, chosen on the development split by a pre-registered rule (smallest passing candidates) before the sealed set was opened. They were this checkpoint and ThakiCloud/eg2-w6-text-w4-media.

Mixed-bit superiority is not established. At the same size, the uniform control ThakiCloud/eg2-w6-text-w4-media scored 0.3218 / 0.2676 (image / video R@1) against this checkpoint's 0.3204 / 0.2636. The development-split video lead of +1.68 points did not replicate on the sealed set (-0.40 points). No paired test between the two 550 MB checkpoints was registered, so neither is claimed to be better than the other.

Video by source, reported only (the verdict uses the pooled set):

Development split (used for selection, not independent)

These numbers chose the candidates and the bit allocation, so they are not independent evidence. Image: 1,609 queries, 340 clusters. Video: 537 Korean queries, 186 clusters. BF16 R@1 0.3443 / 0.3501; this checkpoint 0.3381 / 0.3277 (image / video). On 537 video queries, non-inferiority to BF16 could not be decided; the sealed set (2.3× larger) decided it.

Audio check on development sets (2026-10-20, not sealed)

Same sentence-transformers stack as above: audio mono, resampled to 16 kHz, first 30 s, no document prompt, queries with SearchQuery, vectors renormalized in fp32. Each model encodes its own queries and audio. The margin is the rule used for the sealed evaluation (δ = 10% of BF16 R@1, audio needs ≥ 600 queries); intervals are 95% cluster bootstraps (2,000 resamples, seed 0). These sets were not used to build or choose this checkpoint, but they are development sets, so this is not a sealed verdict.

• The English sound-description task is hard for BF16 itself (R@1 0.061), so read this as "matches BF16", not as "good at sound search".
• The Korean set is spoken-content retrieval (a summary finds the speech clip), not general sound search.
• Encoding BF16 twice with the same settings gave identical scores (top-1 changed for 0 queries). The audio nondeterminism recorded earlier was on a vLLM serving path, which was not used here.

Scope of the numbers

All numbers come from the sentence-transformers stack shown under Usage. An internal check of BF16 EmbeddingGemma 2 served through vLLM found rank-1 changes on 17.5% of image queries against the same model run in sentence-transformers. Do not mix vectors from different serving stacks in one index, and do not assume these numbers carry over to another stack.

What this saves, and what it does not

A disk and download reduction only. The loader dequantizes every matrix to BF16 at load time, so memory after loading matches BF16 and no speed-up is claimed. Peak load memory and load time were not measured.

Usage

Do not load this repository with SentenceTransformer("…") or AutoModel.from_pretrained("…") directly. The weights are packed integer codes plus group scales; eg2_quant.py in this repository dequantizes them and returns an ordinary SentenceTransformer.

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("ThakiCloud/eg2-mixed-bit-550mb")
sys.path.insert(0, path)
from eg2_quant import load_sentence_transformer

model = load_sentence_transformer(path)          # BF16 on GPU if available

Encode each modality the same way as upstream EmbeddingGemma 2, and normalize after encoding.

import numpy as np
from PIL import Image

unit = lambda x: x / np.linalg.norm(x, axis=-1, keepdims=True)
q = unit(model.encode(["a red circle on a light background"], prompt_name="SearchQuery"))
img = unit(model.encode([{"image": Image.open("photo.jpg")}]))
vid = unit(model.encode([{"video": list_of_pil_frames}]))          # e.g. 8 frames per segment
aud = unit(model.encode([{"audio": waveform_16k}]))                 # mono float32 at 16 kHz, up to 30 s
scores = q @ np.concatenate([img, vid, aud]).T

What this release does not claim

• Audio quality on a sealed set (only development sets were measured), text-only retrieval quality, or quality for all four modalities together.
• That one 550 MB allocation is better than the other, or that mixed precision beats uniform precision in general.
• That sensitivity measured on images predicts sensitivity on video. The sealed result does not support it.
• Run-time memory savings, speed-ups, or results under serving stacks other than sentence-transformers.
• Third-party reproducibility of the numbers: the evaluation data and harness are internal.

Provenance

Sealed result file sha256 c8c50628de1fa1f125445af1c88ed96f98da45631107ea20cf04d0f6389729fc · selection artifact sha256 09811cbda126869e041419c1e4e55b70f6d92034ffb59d3e74580ac21337a70d · pre-registration commit 9ce1058 (internal repository). Weights in this repository were not changed when this card was updated with the sealed results.

About the evaluation data

The image set and the 71866 video queries come from Korean public datasets distributed by AI Hub (aihub.or.kr) under terms that do not allow redistribution. DiDeMo segments are Flickr videos under Creative Commons licenses. None of these were used to train anything, none are included here, and every sealed item passed a near-duplicate audit against training and calibration data before scoring.

License and attribution

Apache License 2.0, inherited from the base model. A copy is included as LICENSE.

Modifications made to the base model: weight matrices were quantized as described above (RTN, no calibration data, no training or fine-tuning). The audio tower's output projection and the audio embedding projection stay BF16.

Base model © Google DeepMind. This is an independently quantized derivative and is not endorsed by Google.