Meditor-1-2.2B-A500M-MoE — First-Generation Sparse MoE (Native)
Meditor-1-2.2B-A500M-MoE is the inaugural sparse Mixture-of-Experts architecture in the Meditor series, expanding Qwen/Qwen2.5-0.5B-Instruct with 1 shared expert (4864) and 22 fine-grained routed experts (1216, top-1 routing) plus 524K sliding+global attention extension.
Universal Edition: Looking for standard dense
qwen2/ GGUF checkpoints? SeeYo-Purple/Meditor-1-500M-Universal.
Meditor Model Series
| Generation & Variant | Hugging Face Repo | Base Backbone | Params (Total / Active) | UniBench-100 Mean (/10) | Pass Rate | VRAM Footprint |
|---|---|---|---|---|---|---|
| Meditor 1 (Native MoE) (This Model) | Yo-Purple/Meditor-1-2.2B-A500M-MoE |
Qwen2.5-0.5B | 2.22B / 0.57B | 2.09 (37/100) · 2.42 (Best M1) | 1.0% (Best M1) | 4.16 GB (BF16) |
| Meditor 1 (Universal) | Yo-Purple/Meditor-1-500M-Universal |
Qwen2.5-0.5B (qwen2) |
0.49B / 0.49B | 2.42 (100/100 BF16) | 1.0% | 0.93 GB / GGUF |
| Meditor 2 (Native HMoE) | Yo-Purple/Meditor-2-3.4B-A1.8B-HMoE |
SmolLM2-1.7B | 3.41B / 1.83B | 3.28 (37/100) · 4.12 (Best M2) | 22.0% (Best M2) | 6.42 GB (BF16) |
| Meditor 2 (Universal) | Yo-Purple/Meditor-2-3.4B-Universal |
SmolLM2-1.7B (qwen2_moe) |
3.41B / 1.83B | 4.12 (100/100 BF16) · 4.00 (Q4_K_M) | 22.0% / 18.0% | 6.38 GB / GGUF |
| Meditor 3 (Native MoE) | Yo-Purple/Meditor-3-4B-MoE |
Qwen3-4B (meditor3) |
4.16B / 4.04B–4.07B | 7.31 (100/100, Rank #1/36) | 69.0% (7.0% Uniq) | 5.43 GiB peak (bf16_low_vram) |
| Meditor 3 (Universal GGUF) | Yo-Purple/Meditor-3-4B-Universal |
Qwen3-4B (k2-horizon) |
4.16B (BF16) · Q8_0 · Q4_K_M |
Compiled GGUF from M3 (7.31 native) |
LM Studio / llama.cpp | 4.37–5.95 GiB (quantized) |
Universal LLM Benchmark (UniBench-100) Cross-Generation Comparison
All benchmark comparisons below come directly from the unified UniBench-100 evaluation suite (100 open-ended synthesis questions across 10 cognitive domains, graded out of 10.0 with pass threshold >= 6.0 by gemini-3.8-flash).
1. Cross-Generation Progression (Full 100/100 Runs)
| Rank (of 36 Runs) | Model Run | Series Generation | Evaluated | Mean Score (/10) | Pass Rate (>=6.0) | Unique Solve Rate | Eval Speed |
|---|---|---|---|---|---|---|---|
| #1 | Meditor-3-4B-MoE (Released) | Meditor 3 (Native) | 100 / 100 | 7.31 | 69.0% | 7.0% | 35.7 tok/s |
| #2 | Meditor-3 Prior Uncontested (BF16) | Meditor 3 (Prior R&D) | 100 / 100 | 4.62 | 30.0% | 1.0% | 35.7 tok/s |
| #5 | Meditor 3 Beta (BF16) | Meditor 3 (Prior R&D) | 100 / 100 | 4.31 | 22.0% | 1.0% | 19.0 tok/s |
| #11 | Meditor 2 Universal (BF16) | Meditor 2 Best | 100 / 100 | 4.12 | 22.0% | 0.0% | 27.5 tok/s |
| #18 | Meditor 2 Universal (Q4_K_M) | Meditor 2 Quantized | 100 / 100 | 4.00 | 18.0% | 0.0% | 141.2 tok/s |
| #29 | Meditor 2 (Native HMoE) | Meditor 2 Native | 37 / 100 | 3.28 | 8.1% | 0.0% | 48.7 tok/s |
| #30 | Meditor 1 Universal (BF16) | Meditor 1 Best | 100 / 100 | 2.42 | 1.0% | 0.0% | 163.4 tok/s |
| #31 | Meditor 1 Universal (Quant) | Meditor 1 Quantized | 100 / 100 | 2.35 | 0.0% | 0.0% | 171.3 tok/s |
| #33 | Meditor 1 (Native MoE) | Meditor 1 Native | 37 / 100 | 2.09 | 2.7% | 0.0% | 141.9 tok/s |
Key Takeaway: Meditor-3-4B-MoE delivers +3.19 points (
+77.4%relative score jump) over the best Meditor 2 run (4.12), +4.89 points (3.02xscore) over the best Meditor 1 run (2.42), and +2.69 points over the strongest earlier Meditor 3 research run (4.62).
2. Per-Domain Score Breakdown (Best Runs of Meditor 1, Meditor 2, and Meditor 3)
| UniBench-100 Domain | Meditor 1 Best (2.42) |
Meditor 2 Best (4.12) |
Meditor 3 Released (7.31) |
Gain vs M2 | M3 Pass Rate |
|---|---|---|---|---|---|
| Critical Thinking & Argument Evaluation | 3.51 | 5.98 | 9.09 | +3.11 |
100.0% |
| Creative & Stylistic Expression | 2.91 | 4.95 | 8.38 | +3.43 |
90.0% |
| Humanities, Philosophy & Ethics | 2.30 | 4.91 | 8.35 | +3.44 |
80.0% |
| Physical & Life Sciences | 1.68 | 3.38 | 7.41 | +4.03 |
80.0% |
| Mathematics & Quantitative Estimation | 2.36 | 1.99 | 7.27 | +5.28 |
60.0% |
| Factuality & Hallucination Resistance | 1.32 | 5.56 | 7.09 | +1.53 |
60.0% |
| Real-World Tactical Strategy & Decision-Making | 2.10 | 3.72 | 6.98 | +3.26 |
60.0% |
| Instruction Following & Strict Constraints | 3.65 | 4.62 | 6.82 | +2.20 |
50.0% |
| Coding & Software Engineering | 2.15 | 3.26 | 6.46 | +3.20 |
70.0% |
| Logic & Abstract Reasoning | 2.19 | 2.82 | 5.27 | +2.45 |
40.0% |
3. Full Leaderboard Landscape
Architecture Specifications (Meditor-1-2.2B-A500M-MoE)
| Architectural Parameter | Value |
|---|---|
| Model Class | MeditorForCausalLM (meditor, custom Hugging Face auto_map) |
| Backbone Origin | Qwen/Qwen2.5-0.5B-Instruct (24 layers, d_model=896, 14Q / 2KV heads) |
| Attention Layout | 20 sliding-window (2048) + 4 global layers (524,288 context) |
| MoE FFN Layout | 1 Shared Expert (4864) + 22 Routed Experts (1216, top-1 per token) |
| Parameters | 2,220,330,904 Total · 572,952,448 Active (25.8%) |
| BF16 Footprint | 4.39 GB (model.safetensors) · 4.16 GB BF16 VRAM |
Local Hardware & Probe Evaluations
| Evaluation Benchmark | Observed Result |
|---|---|
UniBench-100 (Meditor-1 Native) |
2.09 / 10 (2.7% pass on 37/100 partial eval @ 141.9 tok/s) · Universal BF16 best M1 run reached 2.42 / 10 (100/100) |
| WikiText-2 Perplexity | 8.35 (loss 2.1227) |
Commonsense (12 Questions) |
50.0% (6 / 12) |
Arithmetic (10 Questions) |
60.0% raw · 100.0% with tool assist |
PyTorch Speed (seq_256) |
18.52 tok/s |
Quickstart
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Yo-Purple/Meditor-1-2.2B-A500M-MoE"
tok = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")
inputs = tok("Explain photosynthesis:", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=96)[0], skip_special_tokens=True))
License
Released under Apache 2.0. Built on Qwen/Qwen2.5-0.5B-Instruct.