tiny-embed-61m
Model Details | Usage | Evaluation | Contact | License
tiny-embed-61m maps Chinese text to a low-dimensional dense vector which can be used for
tasks like retrieval, classification, clustering, or semantic search. It is a compact
(61.4M parameters, 768-dim) model distilled from larger open Chinese embedding models, and
it is designed to run cheaply on CPU or a single consumer GPU while keeping retrieval
quality close to bge-base-zh-v1.5 (102M).
No query instruction is required — the model is trained and evaluated without any prefix. Passages and queries both take the original text.
************* 🌟Highlights🌟 *************
- 61.4M parameters, 768-dim output, 512 tokens max length.
- 0.6058 average on
MTEB(cmn, v1)(31 tasks, no instruction), vs.bge-small-zh-v1.50.5734 andbge-base-zh-v1.50.6270 measured on the same harness. - Pure
sentence-transformerspipeline:Transformer → Pooling(mean) → Normalize → Dense(512→768) → Normalize.
Model Details
| Item | Value |
|---|---|
| Model type | Dense text embedding (bi-encoder) |
| Parameters | 61,425,664 (backbone) + 393,216 (projection head) |
| Embedding dimension | 768 |
| Backbone hidden size | 512 |
| Layers | 12 |
| Attention heads | 8 |
| Pooling | mean, then L2 normalize |
| Projection head | linear 512 → 768, bias-free, identity activation (part of the trained model, not a post-processing step) |
| Max sequence length | 512 |
| Tokenizer | Chinese BERT vocabulary (21,128 tokens) |
| Similarity | cosine (embeddings are L2-normalized) |
| Precision | fp16 / bf16 supported |
The model was distilled from thenlper/gte-base-zh at its native 768 dimensions, so the
student and the teacher share the same embedding space without dimensionality reduction.
Training data & benchmark overlap
Training data are Chinese retrieval, STS and instruction corpora (webqa, cmedqa, chinese_agriculture_qa, pawsx, wiki_zh, fineweb_edu_zh, clean_passages, alpaca_zh) plus synthesized queries. Benchmark-related subsets are explicitly declared in the MTEB model metadata:
| Status | Benchmark tasks |
|---|---|
| Declared (their corpora appear in training data) | PAWSX, CmedqaRetrieval, CMedQAv1-reranking, CMedQAv2-reranking |
| Excluded (corpora removed from training data → zero-shot) | T2Retrieval, T2Reranking (T2Ranking family), DuRetrieval (DuReader), ATEC, BQ, LCQMC, STSB |
Usage
Using Sentence-Transformers
pip install -U sentence-transformers
Two things are non-optional when loading this checkpoint:
trust_remote_code=True— the backbone is shipped astiny_st_module.pyinside this repository.- Keep the
3_Densemodule (512 → 768). It is part of the trained model, not a post-processing step: training usednormalize(mean_pool(hidden)) → Dense(512→768) → normalize, and the published scores are measured on those 768-d outputs. Loading only the backbone and skipping3_Denseyields 512-d embeddings that do not reproduce the scores below.
from sentence_transformers import SentenceTransformer
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]
model = SentenceTransformer("dynasty-ai/tiny-embed-61m", trust_remote_code=True)
embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity) # shape: (2, 768)
For short-query to long-passage retrieval, encode queries and passages the same way — no instruction is needed on either side:
queries = ["停经半年后又现点红是什么原因", "中医治疗失眠的方法"]
passages = ["样例文档-1", "样例文档-2"]
q_embeddings = model.encode(queries, normalize_embeddings=True)
p_embeddings = model.encode(passages, normalize_embeddings=True)
scores = q_embeddings @ p_embeddings.T
Using Huggingface Transformers
The checkpoint also loads with transformers, but you must reproduce the pooling and the
projection head yourself:
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("dynasty-ai/tiny-embed-61m", trust_remote_code=True)
model = AutoModel.from_pretrained("dynasty-ai/tiny-embed-61m", trust_remote_code=True)
model.eval()
sentences = ["样例数据-1", "样例数据-2"]
encoded_input = tokenizer(sentences, padding=True, truncation=True,
max_length=512, return_tensors="pt")
with torch.no_grad():
token_embeddings = model(**encoded_input)["token_embeddings"]
mask = encoded_input["attention_mask"].unsqueeze(-1).to(token_embeddings.dtype)
pooled = (token_embeddings * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
pooled = F.normalize(pooled, p=2, dim=1) # mean pooling
head = torch.nn.Linear(512, 768, bias=False) # load from 3_Dense/pytorch_model.bin
head.load_state_dict({"weight": torch.load("3_Dense/pytorch_model.bin",
map_location="cpu")["linear.weight"]})
embeddings = F.normalize(head(pooled), p=2, dim=1)
print(embeddings.shape) # (2, 768)
For most users the sentence-transformers route above is simpler and bit-identical.
Evaluation
MTEB(cmn, v1) — Chinese MTEB, 31 tasks, no instruction prefix, cosine similarity,
evaluated with mteb 1.39.7. Scores are the average of the standard task metric per task
group.
| Model | Dimension | Average | Retrieval (8) | Reranking (4) | Clustering (4) | Classification (9) | PairClassification (2) | STS (4) |
|---|---|---|---|---|---|---|---|---|
| tiny-embed-61m | 768 | 60.58 | 63.50 | 66.23 | 49.61 | 62.10 | 68.78 | 52.56 |
| bge-small-zh-v1.5 | 512 | 57.34 | 61.74 | 61.24 | 44.17 | 59.91 | 65.56 | 47.91 |
| bge-base-zh-v1.5 | 768 | 62.70 | 69.40 | 64.87 | 47.30 | 64.21 | 73.17 | 53.90 |
| gte-base-zh | 768 | 65.72 | 71.73 | 67.10 | 53.63 | 66.52 | 74.75 | 58.13 |
Same-harness observations:
- vs.
bge-small-zh-v1.5(24M): +3.24 average, better in every task group. - vs.
bge-base-zh-v1.5(102M, 1.7× parameters): −2.12 average — competitive on reranking / clustering / STS, still behind on retrieval.
Retrieval detail (T2Ranking dev, nDCG@10 / recall@10, no instruction): T2Retrieval nDCG@10 78.21 / recall@10 90.10; MMarcoRetrieval 65.29.
Note on benchmark tasks:
T2Retrieval/T2Rerankingare zero-shot for this model (T2Ranking corpora were excluded from training). Subsets that do appear in training data are listed in the MTEB model metadata.
Contact
Questions, bug reports and suggestions are welcome:
- Open a discussion on the Community tab of this model page (dynasty-ai/tiny-embed-61m) — this is the model "issue tracker"; please include library versions, hardware and a minimal reproduction.
- Or reach the maintainers of the dynasty-ai organization via their Hugging Face profile: https://huggingface.co/dynasty-ai.
License
tiny-embed-61m is licensed under the Apache-2.0 License.
It is produced by distillation from the MIT-licensed model thenlper/gte-base-zh; the
copyright notice and permission notice of the upstream work apply to the corresponding
upstream model. Training corpora retain their original licenses. The released weights can
be used for commercial purposes free of charge.