tiny-embed-27m
Model Details | Usage | Evaluation | Contact | License
tiny-embed-27m maps Chinese text to a low-dimensional dense vector which can be used for
tasks like retrieval, classification, clustering, or semantic search. It is a compact
(27.6M parameters, 512-dim) model distilled from larger open Chinese embedding models. Its
backbone shares 4 unique layers applied 16 times (recursive depth), which keeps the
parameter count low at inference-friendly latency.
No query instruction is required — the model is trained and evaluated without any prefix. Passages and queries both take the original text.
************* 🌟Highlights🌟 *************
- 27.6M parameters, 512-dim output, 512 tokens max length.
- 0.5963 average on
MTEB(cmn, v1)(31 tasks, no instruction), vs.bge-small-zh-v1.5(24M) 0.5734 measured on the same harness. - Standard
sentence-transformerspipeline:Transformer → Pooling(mean) → Normalize(no extra projection layer).
Model Details
| Item | Value |
|---|---|
| Model type | Dense text embedding (bi-encoder) |
| Parameters | 27,599,872 |
| Embedding dimension | 512 |
| Backbone hidden size | 512 |
| Layers | 4 unique, repeated 16× (weight-shared recursive depth) |
| Attention heads | 8 |
| Pooling | mean, then L2 normalize |
| Max sequence length | 512 |
| Tokenizer | Chinese BERT vocabulary (21,128 tokens) |
| Similarity | cosine (embeddings are L2-normalized) |
| Precision | fp16 / bf16 supported |
The model was distilled from a three-teacher vector soup built from
BAAI/bge-small-zh-v1.5, BAAI/bge-base-zh-v1.5 and thenlper/gte-base-zh
(all MIT-licensed).
Training data & benchmark overlap
Training data are Chinese retrieval, STS and instruction corpora (webqa, cmedqa, chinese_agriculture_qa, pawsx, wiki_zh, fineweb_edu_zh, clean_passages, alpaca_zh) plus synthesized queries. Benchmark-related subsets are explicitly declared in the MTEB model metadata:
| Status | Benchmark tasks |
|---|---|
| Declared (their corpora appear in training data) | PAWSX, CmedqaRetrieval, CMedQAv1-reranking, CMedQAv2-reranking |
| Excluded (corpora removed from training data → zero-shot) | T2Retrieval, T2Reranking (T2Ranking family), DuRetrieval (DuReader), ATEC, BQ, LCQMC, STSB |
Usage
Using Sentence-Transformers
pip install -U sentence-transformers
trust_remote_code=True is required: the backbone is shipped as tiny_st_module.py
inside this repository. Unlike tiny-embed-61m, this checkpoint has no projection head: the pipeline is exactly Transformer → Pooling(mean) → Normalize, so the 512-d mean-pooled vector is the final embedding.
from sentence_transformers import SentenceTransformer
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]
model = SentenceTransformer("dynasty-ai/tiny-embed-27m", trust_remote_code=True)
embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity) # shape: (2, 512)
Short-query to long-passage retrieval — encode both sides identically, no instruction:
queries = ["停经半年后又现点红是什么原因", "中医治疗失眠的方法"]
passages = ["样例文档-1", "样例文档-2"]
q_embeddings = model.encode(queries, normalize_embeddings=True)
p_embeddings = model.encode(passages, normalize_embeddings=True)
scores = q_embeddings @ p_embeddings.T
Using Huggingface Transformers
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("dynasty-ai/tiny-embed-27m", trust_remote_code=True)
model = AutoModel.from_pretrained("dynasty-ai/tiny-embed-27m", trust_remote_code=True)
model.eval()
sentences = ["样例数据-1", "样例数据-2"]
encoded_input = tokenizer(sentences, padding=True, truncation=True,
max_length=512, return_tensors="pt")
with torch.no_grad():
token_embeddings = model(**encoded_input)["token_embeddings"]
mask = encoded_input["attention_mask"].unsqueeze(-1).to(token_embeddings.dtype)
pooled = (token_embeddings * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
embeddings = F.normalize(pooled, p=2, dim=1) # mean pooling + L2 normalize
print(embeddings.shape) # (2, 512)
Evaluation
MTEB(cmn, v1) — Chinese MTEB, 31 tasks, no instruction prefix, cosine similarity,
evaluated with mteb 1.39.7. Scores are the average of the standard task metric per task
group.
| Model | Dimension | Average | Retrieval (8) | Reranking (4) | Clustering (4) | Classification (9) | PairClassification (2) | STS (4) |
|---|---|---|---|---|---|---|---|---|
| tiny-embed-27m | 512 | 59.63 | 62.36 | 65.04 | 48.61 | 61.22 | 68.64 | 51.71 |
| bge-small-zh-v1.5 | 512 | 57.34 | 61.74 | 61.24 | 44.17 | 59.91 | 65.56 | 47.91 |
| m3e-small | — | 52.75 | — | — | — | — | — | — |
| bge-base-zh-v1.5 | 768 | 62.70 | 69.40 | 64.87 | 47.30 | 64.21 | 73.17 | 53.90 |
Same-harness observations:
- vs.
bge-small-zh-v1.5(24M): +2.29 average, better in every task group. - vs.
bge-base-zh-v1.5(102M, 3.7× parameters): −3.07 average; the gap is concentrated in retrieval (−7.04) and pair classification (−4.53).
Retrieval detail (T2Ranking dev, no instruction): recall@10 89.56; T2Retrieval zero-shot (T2Ranking corpora excluded from training).
Note on benchmark tasks:
T2Retrieval/T2Rerankingare zero-shot for this model. Subsets that do appear in training data are listed in the MTEB model metadata.
Contact
Questions, bug reports and suggestions are welcome:
- Open a discussion on the Community tab of this model page (dynasty-ai/tiny-embed-27m) — this is the model "issue tracker"; please include library versions, hardware and a minimal reproduction.
- Or reach the maintainers of the dynasty-ai organization via their Hugging Face profile: https://huggingface.co/dynasty-ai.
License
tiny-embed-27m is licensed under the Apache-2.0 License.
It is produced by distillation from the MIT-licensed models BAAI/bge-small-zh-v1.5,
BAAI/bge-base-zh-v1.5 and thenlper/gte-base-zh; the copyright notices and permission
notices of those upstream works apply to the corresponding upstream models. Training
corpora retain their original licenses. The released weights can be used for commercial
purposes free of charge.