MiniMind-Embedding-MoE

A Mixture-of-Experts (MoE) text embedding model built on the MiniMind-3-MoE backbone (198M total / 64M active params), with the training pipeline fully aligned with Qwen3-Embedding. This is the MoE counterpart of MiniMind-Embedding-Dense.

📌 Introduction

This model uses the same Qwen3-Embedding recipe as the Dense version (causal attention + last-token pooling + InfoNCE with false-negative masking + MRL), but swaps the feed-forward layers for a 4-expert top-1 MoE. The goal is to study whether MoE architecture helps embedding quality at the small scale.

Core techniques

🏗️ Model Details

🎓 Training Details (3 Stages)

MoE Stage 1 (left) + Stage 2 (right), 3 epochs.

Dense vs MoE Training Loss

Note: MoE loss is consistently higher than Dense. This is expected — MoE routes tokens to different experts, making the InfoNCE contrastive task harder (each expert sees only a subset of the sequence's tokens). The higher train loss doesn't directly translate to worse embeddings, but the evaluation shows MoE underperforms Dense at this scale.

📊 Evaluation

C-MTEB STS (Spearman correlation)

Key Finding: MoE underperforms Dense at small scale

💡 Surprising result: the MoE model (198M total / 64M active) underperforms the Dense model (64M) by 0.056 average STS. This is a counterintuitive but important finding:

1. MoE fragmentation: with only 4 experts and top-1 routing, each expert sees only ~25% of tokens. For embedding (where the whole sentence's representation matters), this fragmentation hurts — no single expert gets a complete view of the sentence.
2. Active params are the same: MoE activates 64M params per token (same as Dense's 64M), so it's not "bigger" in terms of compute — it's just more fragmented.
3. Routing overhead: the router must learn to dispatch tokens, adding an optimization burden that competes with learning good representations.

Takeaway: for small embedding models, Dense > MoE. MoE benefits emerge at larger scales (Qwen3-Embedding uses MoE only at 4B+ params) where each expert is large enough to be expressive on its own.

🚀 Usage

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("Muzian/minimind-embedding-moe")

documents = ["天空呈蓝色是因为阳光散射。", "红烧肉的做法是..."]
doc_embs = model.encode(documents, normalize_embeddings=True)

queries = ["天空为什么是蓝色的"]
query_embs = model.encode(queries, prompt_name="query", normalize_embeddings=True)

scores = cos_sim(query_embs, doc_embs)

Recommendation: for most use cases, prefer MiniMind-Embedding-Dense — it's smaller, faster, and scores higher. This MoE model is primarily for research/comparison.

⚠️ Limitations

1. Same backbone limitations as Dense (vocab=6400, limited Stage-1 data)
2. MoE underperforms Dense at this scale — see analysis above
3. 198M params but only 64M active → inference is slower than Dense without quality gain

🙏 Acknowledgements

• Base: MiniMind by @jingyaogong
• Route: Qwen3-Embedding
• Data: T2Ranking
• Eval: MTEB

📄 License

Apache 2.0