MiniMind-Rerank-MoE
A Mixture-of-Experts cross-encoder reranker built on the MiniMind-3-MoE backbone (198M total / 64M active). The MoE counterpart of MiniMind-Rerank-Dense.
📌 How It Works
Same pointwise yes/no design as the Dense version: given a query and document, the model predicts whether the next token is 是 (relevant) or 否 (irrelevant). The softmax probability of 是 is the relevance score.
📊 Evaluation
C-MTEB/T2Reranking (MAP@10, no data leakage)
| Model | MAP@10 | Accuracy |
|---|---|---|
| Rerank-Dense (64M) | 0.915 | 85% |
| Rerank-MoE (198M-A64M) | 0.604 | 88% |
Key Finding: MoE classification > Dense, but ranking < Dense
💡 Counterintuitive: MoE's training accuracy (88%) is higher than Dense (85%), yet its MAP@10 (0.604) is far lower (Dense: 0.915). Why?
- Pointwise ≠ Ranking: MoE excels at binary classification (yes/no), but ranking requires fine-grained discrimination between "very relevant" vs "somewhat relevant." MoE's top-1 routing fragments the sequence across experts, making it harder to form a coherent global relevance signal.
- Consistent with embedding findings: just as MoE underperforms Dense on STS (0.422 vs 0.478), the MoE architecture hurts representation quality at small scales.
Recommendation: for reranking, use MiniMind-Rerank-Dense — smaller, faster, and scores 50% higher.
🎓 Training
| Config | Value |
|---|---|
| Training data | T2Reranking, 80k pairs (80/20 split, no leakage) |
| Epochs | 3 |
| Batch size | 16 |
| Learning rate | 1e-5 (cosine) |
| Final loss | 0.28 |
| Final accuracy | 88% |
🏗️ Model Details
| Item | Value |
|---|---|
| Architecture | MiniMind-3-MoE (4 experts, top-1) |
| Total params | 198M |
| Active params | 64M |
| Scoring | pointwise yes/no |
| Target tokens | 是 (357) / 否 (1332) |
🙏 Acknowledgements
- Base: MiniMind by @jingyaogong
- Route: Qwen3-Reranker
- Data: C-MTEB/T2Reranking
📄 License
Apache 2.0