EmbeddingGemma-2 text · 0.8 BPW
EmbeddingGemma-2 quantized to 0.79982 bits per weight, with CPU inference and support for text.
| Property | Value |
|---|---|
| Checkpoint size | 27.09 MB |
| Weight size reduction vs BF16 | 95.00% (about 20.0× smaller) |
| Parameters in this variant | 271,002,648 |
| Output dimensions | 768; optional 512, 256 or 128 |
| Similarity | Cosine |
Quantization
Uses binary latent-factor quantization inspired by LittleBit: bit-packed factors, FP16 scales and selected higher-precision weights. A rank-4 query adapter is included for SearchQuery. The complete weight file, including the adapter, scales and dense weights, stays below 0.8 BPW. The size reduction describes stored weights; runtime memory includes activations and unpacked factors.
Text retrieval results
FiQA dense retrieval with cosine similarity and a 256-token limit. Validation uses 129 queries; test uses 131 queries. Each split has 2,048 candidate documents, including the relevant documents and sampled distractors. These are reduced-corpus measurements.
| Encoder / split | Recall@1 | Recall@10 | Recall@50 | nDCG@10 | MRR@10 |
|---|---|---|---|---|---|
| Base · test | 36.82% | 68.10% | 89.04% | 0.6343 | 0.7377 |
| ONNX INT8 · test | 36.82% | 68.10% | 89.04% | 0.6345 | 0.7379 |
| 0.8 BPW + R4 · test | 22.72% | 55.53% | 79.20% | 0.4601 | 0.5410 |
| 0.8 BPW + R4 · validation | 27.05% | 58.27% | 81.50% | 0.4974 | 0.5769 |
Recall@10 loss: 18.46% relative to the base on test (12.57 percentage points); 24.08% on validation (base: 76.75%). Against ONNX INT8, the relative losses are 18.46% on test and 23.85% on validation (INT8: 76.51%). Test metrics were reproduced on CPU.
The query adapter was fitted on FiQA development data. These results measure text retrieval only. Image, audio and video inputs, where supported, have been verified for CPU inference; their retrieval Recall has not been evaluated. Quality outside FiQA and for truncated embeddings has not been measured here.
Usage
Download the repository with the Hugging Face CLI, enter its directory and install requirements.txt. For CPU PyTorch, install the CPU wheels before the remaining requirements:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt
from modeling_littlebit import EmbeddingGemma2LittleBit
model = EmbeddingGemma2LittleBit.from_pretrained(".", device="cpu")
query = model.encode("What is compound interest?", prompt_name="SearchQuery")
document = model.encode(
"Compound interest includes interest earned on accumulated interest.",
prompt_name="Document",
)
score = float(query[0] @ document[0])
Use the included loader for this packed format. Text lists support batch_size.
Base model: Google DeepMind · EmbeddingGemma-2 · Apache 2.0.