MiniMind-Embedding-Dense

A text embedding model built on the MiniMind-3 small language model (64M params), with the training pipeline fully aligned with Qwen3-Embedding. Part of the minimind-embedding project.

📌 Introduction

This model turns the 64M MiniMind-3 causal language model into a text encoder using three key techniques:

1. Last-Token Pooling

💡 Think of a sentence as a queue of people. The last person (EOS) has "heard" everyone before them through causal attention, so they alone can summarize the whole sentence.

Instead of BERT-style bidirectional attention + mean pooling, we keep causal (unidirectional) attention and take the hidden state at the appended EOS token. The last token aggregates information from all preceding tokens via causal attention, naturally serving as a sentence summary. This maximizes reuse of pretrained weights.

2. InfoNCE + False-Negative Masking

💡 Contrastive learning is like a "find your friend" game: push the correct match closer, push strangers further away. But what if a "stranger" is actually your friend (false negative)? We detect and mask them!

The core loss is InfoNCE (contrastive). Qwen3-Embedding adds a false-negative mask: if a candidate negative is more similar to the query than the positive (beyond margin=0.1), it's likely a false negative and gets masked out.

3. Matryoshka Representation Learning (MRL)

💡 Like Russian nesting dolls — the big doll (768-dim) contains smaller dolls (512/256/128/64-dim), each fully functional at its size.

Trained so the vector works at any truncated dimension. Use 64-dim for fast coarse retrieval, 768-dim for precise re-ranking.

🏗️ Model Details

🎓 Training Details (3 Stages)

Stage 1 (left): 1.16→0.43. Stage 2 (right): 1.47→0.72, 3 epochs with epoch boundaries marked.

📊 Evaluation

Honest note: scores are bounded by the small vocab (6400) and limited Stage-1 data (90k vs Qwen3's 150M). The value is the complete, reproducible Qwen3-Embedding pipeline.

🚀 Usage

sentence-transformers

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("Muzian/minimind-embedding-dense")

documents = ["天空呈蓝色是因为阳光散射。", "红烧肉的做法是..."]
doc_embs = model.encode(documents, normalize_embeddings=True)

queries = ["天空为什么是蓝色的"]
query_embs = model.encode(queries, prompt_name="query", normalize_embeddings=True)

scores = cos_sim(query_embs, doc_embs)

Matryoshka (truncate dimensions)

# Use only first 256 dims (smaller index, faster search)
emb_256 = embedding[:, :256]
emb_256 = F.normalize(emb_256, p=2, dim=1)

⚠️ Limitations

1. Small vocabulary (6400) — weak Chinese compression, primary bottleneck
2. Limited Stage-1 data (90k vs Qwen3's 150M) — scaling is the top improvement direction
3. SLERP merging shows minimal gain on small models (Stage2 ≈ Stage3)

🙏 Acknowledgements

• Base: MiniMind by @jingyaogong
• Route: Qwen3-Embedding
• Data: T2Ranking
• Eval: MTEB

📄 License

Apache 2.0