Meditor-2-3.4B-A1.8B-HMoE — Hierarchical Mixture-of-Experts (Native)
Meditor-2-3.4B-A1.8B-HMoE is the second-generation native Hierarchical Mixture-of-Experts (HMoE) model in the Meditor series, upgrading HuggingFaceTB/SmolLM2-1.7B-Instruct with a dense shared core and 2-level hierarchical domain routing (3 macro domain routers -> 5 sub-experts each = 15 routed specialists, top-1 active).
Universal Edition: Looking for standard
Qwen2MoeForCausalLMand GGUF formats? SeeYo-Purple/Meditor-2-3.4B-Universal.
Meditor Model Series
| Generation & Variant | Hugging Face Repo | Base Backbone | Params (Total / Active) | UniBench-100 Mean (/10) | Pass Rate | VRAM Footprint |
|---|---|---|---|---|---|---|
| Meditor 1 (Native MoE) | Yo-Purple/Meditor-1-2.2B-A500M-MoE |
Qwen2.5-0.5B | 2.22B / 0.57B | 2.09 (37/100) · 2.42 (Best M1) | 1.0% (Best M1) | 4.16 GB (BF16) |
| Meditor 1 (Universal) | Yo-Purple/Meditor-1-500M-Universal |
Qwen2.5-0.5B (qwen2) |
0.49B / 0.49B | 2.42 (100/100 BF16) | 1.0% | 0.93 GB / GGUF |
| Meditor 2 (Native HMoE) (This Model) | Yo-Purple/Meditor-2-3.4B-A1.8B-HMoE |
SmolLM2-1.7B | 3.41B / 1.83B | 3.28 (37/100) · 4.12 (Best M2) | 22.0% (Best M2) | 6.42 GB (BF16) |
| Meditor 2 (Universal) | Yo-Purple/Meditor-2-3.4B-Universal |
SmolLM2-1.7B (qwen2_moe) |
3.41B / 1.83B | 4.12 (100/100 BF16) · 4.00 (Q4_K_M) | 22.0% / 18.0% | 6.38 GB / GGUF |
| Meditor 3 (Native MoE) | Yo-Purple/Meditor-3-4B-MoE |
Qwen3-4B (meditor3) |
4.16B / 4.04B–4.07B | 7.31 (100/100, Rank #1/36) | 69.0% (7.0% Uniq) | 5.43 GiB peak (bf16_low_vram) |
| Meditor 3 (Universal GGUF) | Yo-Purple/Meditor-3-4B-Universal |
Qwen3-4B (k2-horizon) |
4.16B (BF16) · Q8_0 · Q4_K_M |
Compiled GGUF from M3 (7.31 native) |
LM Studio / llama.cpp | 4.37–5.95 GiB (quantized) |
Universal LLM Benchmark (UniBench-100) Cross-Generation Comparison
All benchmark comparisons below come directly from the unified UniBench-100 evaluation suite (100 open-ended synthesis questions across 10 cognitive domains, graded out of 10.0 with pass threshold >= 6.0 by gemini-3.8-flash).
1. Cross-Generation Progression (Full 100/100 Runs)
| Rank (of 36 Runs) | Model Run | Series Generation | Evaluated | Mean Score (/10) | Pass Rate (>=6.0) | Unique Solve Rate | Eval Speed |
|---|---|---|---|---|---|---|---|
| #1 | Meditor-3-4B-MoE (Released) | Meditor 3 (Native) | 100 / 100 | 7.31 | 69.0% | 7.0% | 35.7 tok/s |
| #2 | Meditor-3 Prior Uncontested (BF16) | Meditor 3 (Prior R&D) | 100 / 100 | 4.62 | 30.0% | 1.0% | 35.7 tok/s |
| #5 | Meditor 3 Beta (BF16) | Meditor 3 (Prior R&D) | 100 / 100 | 4.31 | 22.0% | 1.0% | 19.0 tok/s |
| #11 | Meditor 2 Universal (BF16) | Meditor 2 Best | 100 / 100 | 4.12 | 22.0% | 0.0% | 27.5 tok/s |
| #18 | Meditor 2 Universal (Q4_K_M) | Meditor 2 Quantized | 100 / 100 | 4.00 | 18.0% | 0.0% | 141.2 tok/s |
| #29 | Meditor 2 (Native HMoE) | Meditor 2 Native | 37 / 100 | 3.28 | 8.1% | 0.0% | 48.7 tok/s |
| #30 | Meditor 1 Universal (BF16) | Meditor 1 Best | 100 / 100 | 2.42 | 1.0% | 0.0% | 163.4 tok/s |
| #31 | Meditor 1 Universal (Quant) | Meditor 1 Quantized | 100 / 100 | 2.35 | 0.0% | 0.0% | 171.3 tok/s |
| #33 | Meditor 1 (Native MoE) | Meditor 1 Native | 37 / 100 | 2.09 | 2.7% | 0.0% | 141.9 tok/s |
Key Takeaway: Meditor-3-4B-MoE delivers +3.19 points (
+77.4%relative score jump) over the best Meditor 2 run (4.12), +4.89 points (3.02xscore) over the best Meditor 1 run (2.42), and +2.69 points over the strongest earlier Meditor 3 research run (4.62).
2. Per-Domain Score Breakdown (Best Runs of Meditor 1, Meditor 2, and Meditor 3)
| UniBench-100 Domain | Meditor 1 Best (2.42) |
Meditor 2 Best (4.12) |
Meditor 3 Released (7.31) |
Gain vs M2 | M3 Pass Rate |
|---|---|---|---|---|---|
| Critical Thinking & Argument Evaluation | 3.51 | 5.98 | 9.09 | +3.11 |
100.0% |
| Creative & Stylistic Expression | 2.91 | 4.95 | 8.38 | +3.43 |
90.0% |
| Humanities, Philosophy & Ethics | 2.30 | 4.91 | 8.35 | +3.44 |
80.0% |
| Physical & Life Sciences | 1.68 | 3.38 | 7.41 | +4.03 |
80.0% |
| Mathematics & Quantitative Estimation | 2.36 | 1.99 | 7.27 | +5.28 |
60.0% |
| Factuality & Hallucination Resistance | 1.32 | 5.56 | 7.09 | +1.53 |
60.0% |
| Real-World Tactical Strategy & Decision-Making | 2.10 | 3.72 | 6.98 | +3.26 |
60.0% |
| Instruction Following & Strict Constraints | 3.65 | 4.62 | 6.82 | +2.20 |
50.0% |
| Coding & Software Engineering | 2.15 | 3.26 | 6.46 | +3.20 |
70.0% |
| Logic & Abstract Reasoning | 2.19 | 2.82 | 5.27 | +2.45 |
40.0% |
3. Full Leaderboard Landscape
Architecture Specifications (Meditor-2-3.4B-A1.8B-HMoE)
| Architectural Parameter | Value |
|---|---|
| Model Class | Meditor2ForCausalLM (meditor2, custom Hugging Face auto_map) |
| Backbone Origin | HuggingFaceTB/SmolLM2-1.7B-Instruct (24 layers, d_model=2048, 32Q / 32KV heads, d_head=64) |
| Expert Structure | 1 Shared Dense FFN (8192) + 3 Macro Domains x 5 Sub-Experts = 15 Routed Experts (768, top-1) |
| Parameters | 3,410,954,264 Total · 1,825,507,328 Active (53.52%) |
| Context Extension | 524,288 tokens (rope_theta=130,000.0) |
| BF16 Storage & VRAM | 6.35 GB (model.safetensors) · 6.42 GB allocated BF16 VRAM |
Local Hardware & Domain Probe Evaluations
| Evaluation Benchmark | Observed Result |
|---|---|
UniBench-100 (Meditor-2 Native) |
3.28 / 10 (8.1% pass on 37/100 partial eval @ 48.7 tok/s) · Universal BF16 best M2 run reached 4.12 / 10 (100/100) |
| WikiText-2 Perplexity | Improved 7.35 -> 3.48 post-SFT |
ARC Subset (12 Questions) |
100.0% (12 / 12) |
Arithmetic (10 Questions) |
70.0% unforced · 80.0% forced (100.0% with calculator tool) |
| Python / Dialogue Suites | 80.0% (90.0% routed) on both Python & Dialogue probes |
| Router Routing Accuracy | 100.0% macro & 100.0% micro routing alignment |
| PyTorch Generation Throughput | 12.08 tok/s (seq_256 BF16) |
Quickstart
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Yo-Purple/Meditor-2-3.4B-A1.8B-HMoE"
tok = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")
inputs = tok("Explain entropy concisely:", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(tok.decode(out[0], skip_special_tokens=True))
License
Released under Apache 2.0. Built on HuggingFaceTB/SmolLM2-1.7B-Instruct.