🇰🇷 한국어 카드: MODEL_CARD.ko.md

EG2-Text-Q4

A 4-bit quantized text tower of EmbeddingGemma 2, built to query an existing BF16 EmbeddingGemma 2 index without re-indexing the corpus.

This is a serialized quantized checkpoint validated for embedding-space compatibility against a frozen BF16 index. Catalog or repository availability does not imply serving or runtime compatibility — see What this release does not claim.

Why a quantized query encoder is a compatibility problem

The point of this checkpoint is not that it is small. It is that it can replace the query side of a retrieval system whose corpus index is already built in BF16, without re-embedding the corpus. That is an asymmetric requirement, and measuring it symmetrically hides the failure mode: a quantized encoder can look healthy when it searches its own embedding space while having drifted away from the original BF16 space.

So every number below is measured asymmetrically — quantized queries against a BF16 corpus index.

Evaluation

Measured on an internal frozen retrieval set that is not publicly released (n = 100 queries, 475-document pool, Korean). Both arms were measured in the same job, on the same GPU, with the same tokenizer and the same pool, and the Q4 arm was re-downloaded from storage so that the file being measured is byte-identical to the file published here (model.safetensors SHA-256 d37db886…55aa673, verified before and after the card files were added).

Paired difference on the same queries: margin −0.0106 [0.0082, 0.0134], neighbor retention −0.064 [0.047, 0.083]. Neither interval contains zero, so the degradation is real rather than noise — but it is also small: Q4 keeps roughly 94% of the BF16 top-10 neighborhood.

Intervals are 95% seed-cluster bootstrap (2000 resamples, clustering unit = seed id). Per-arm intervals overlap heavily because query difficulty dominates the variance; the paired difference is the informative statistic.

Definitions. Margin is the mean gap between a query's positive similarity and its hardest negative, with similarities computed between that arm's query embeddings and the BF16 corpus embeddings. Neighbor retention @10 is the overlap between the top-10 BF16-corpus neighbors retrieved by the arm's query and those retrieved by the BF16 query, divided by 10.

Harness sanity check. The BF16 arm is BF16-vs-BF16 by construction, so its neighbor retention must be exactly 1.000. The job asserts this and aborts otherwise. It measured 1.000.

Lower precision was tested and does not work

A simulated uniform 2-bit configuration of the same text tower collapsed on this evaluation — negative margin and near-total loss of the BF16 top-10 neighborhood. Those figures come from simulated quantization rather than a serialized artifact, so they are not quoted here as product numbers. The conclusion we are comfortable stating: Q4 preserves substantially more BF16-index compatibility than the 2-bit configuration we tested.

A 3-bit variant was also built and is deliberately not published with performance numbers. Its build-to-build variance exceeded the difference we were trying to attribute, so no stable figure can be assigned to it. Reporting a single 3-bit number would have been reporting noise.

Usage

⚠️ AutoModel.from_pretrained on its own will not load this checkpoint. The weights are stored as packed integer codes plus per-group scales, so they have to be dequantized first. eg2_quant.py in this repository does that and then builds a standard EmbeddingGemma 2 text model — after loading, it is an ordinary transformers module.

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("ThakiCloud/eg2-text-q4")
sys.path.insert(0, path)
from eg2_quant import load_quantized_text_model, QUERY_PREFIX, DOC_PREFIX

model, processor = load_quantized_text_model(path)        # device inferred; pass device=... to override

Prompt prefixes must match EmbeddingGemma 2's, or similarities against an existing index will be wrong:

QUERY_PREFIX = "task: search result | query: "
DOC_PREFIX   = "title: none | text: "

Pool with mean pooling and then L2-normalize. If you truncate the 768-dimensional output for Matryoshka use, re-normalize after truncating — slicing and comparing cosines without renormalizing degrades ranking silently.

Vision and audio towers are not present in this file. Text embedding never calls them, so they were removed rather than quantized; this checkpoint is text-only. For multimodal use, run the upstream model.

What this release does not claim

Being explicit, because each of these is a claim we checked and cannot make:

• Not serving-validated. No inference endpoint has been started on this checkpoint. Its status is: registered, artifact-validated (SHA-256 and reload verified), serving not tested.
• No hardware deployment claim. We have not benchmarked it on any target device, so there is no statement here about which GPU or phone it runs on, or at what latency or memory.
• Not independently reproducible. The evaluation data and harness are internal and not released, so the numbers above are internally measured with a frozen in-house protocol. They are not third-party verifiable, and we are not claiming otherwise.
• Not a sub-30MB model. It is ~153MB of weights. Reaching tens of megabytes at this quality would require a distilled student, not quantization.
• No multimodal capability. Text only, by construction.

Limitations

Quality was measured on a Korean retrieval set of 100 queries; other languages and task types are unmeasured, despite the base model supporting 100+ languages. 43 of the 100 queries lost at least one top-10 neighbor relative to BF16, so applications sensitive to exact neighbor identity — deduplication, near-duplicate detection, contamination checks — should validate on their own data before swapping the query encoder. Group-boundary placement affects the result slightly: rebuilding at a different group offset moves the margin by about 0.001, which is why the table reports one build and the comparison above is made against that scale.

Related papers

Two measurement papers sit behind this line's bit allocation. Both quantize EmbeddingGemma-300M, the previous generation; EmbeddingGemma 2 is in neither grid, so read them as the prior that shaped this release rather than as evidence about this checkpoint.

• Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families, arXiv:2609.16391 — the received advice for shrinking a retrieval embedder fails to transfer as stated. The embedding table never emerges as the dominant isolated protection priority in any family, and at INT4/g16 the spread between modules is too small to allocate against.
• Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders, arXiv:2610.09227 — quantization-induced representation drift, the same quantity this card's neighbor-retention numbers measure, used as a label-free sensitivity signal. It orders mixed-precision plans at a macro Spearman of 0.911, but at the main budget it does not beat same-budget uniform precision on sealed held-out embedders.

A third paper concerns calibration rather than allocation: When PTQ Calibration Helps or Hurts: Dose-Driven Collapse and Validation-Gated Abstention in Quantized Text Embedders, arXiv:2610.27397. It fits quantizer parameters on a calibration set at 4-bit for e5-base-v2, BGE-M3 and EmbeddingGemma-300M, and finds the outcome is set by the dose of the update: on EmbeddingGemma-300M the same learning-rate grid spans +1.47 to -71.53 points of in-domain NDCG@10. Its validation-gated accept rule never chose a checkpoint below the uncalibrated baseline (0 harmful selections in 30 decisions), but it is a safety trade, not a free gain: mean oracle regret 1.52 pp. This checkpoint is plain RTN with no calibration data, and EmbeddingGemma 2 is not in that grid.

This checkpoint is uniform 4-bit rather than a sensitivity-allocated mix for the reason in the first paper: no module ordering was stable enough to allocate against.

License and attribution

Apache License 2.0, inherited from the base model. The upstream repository declares apache-2.0 and is not gated, and it contains no LICENSE or NOTICE file, so there are no upstream notices to carry forward. A copy of the license is included here as LICENSE.

Modifications made to the base model: the text backbone's linear layers and the token embedding matrix were quantized to 4 bits (RTN, asymmetric, group size 64 along the input dimension, fp16 scale and zero-point with a 6.1e-5 floor); the vision and audio towers were removed; no weights were trained or fine-tuned.

Base model © Google DeepMind. This is an independently quantized derivative and is not endorsed by Google.