Meditor-3-4B-MoE is the flagship native PyTorch Mixture-of-Experts architecture in the Meditor series, built on an intact 4.02B Qwen3-4B shared dense core paired with 8 specialist rank-8 LoRA weight-delta experts (math, logic, code, science, constraints, strategy, creative, crisp) and a 2.90M parameter PromptRouter.
For every user prompt, the PromptRouter inspects pooled hidden states from the shared backbone before generating the first token, selects 1 to 3 specialist domain experts, and dynamically modulates the shared linear projections (q/k/v/o/gate/up/down_proj) across all 36 layers prior to decoding.
Standalone GGUF Edition: Looking for
llama.cpp/ LM Studio compatibility? SeeYo-Purple/Meditor-3-4B-Universal.
| Generation & Variant | Hugging Face Repo | Base Backbone | Params (Total / Active) | UniBench-100 Mean (/10) | Pass Rate | VRAM Footprint |
|---|---|---|---|---|---|---|
| Meditor 1 (Native MoE) | Yo-Purple/Meditor-1-2.2B-A500M-MoE |
Qwen2.5-0.5B | 2.22B / 0.57B | 2.09 (37/100) · 2.42 (Best M1) | 1.0% (Best M1) | 4.16 GB (BF16) |
| Meditor 1 (Universal) | Yo-Purple/Meditor-1-500M-Universal |
Qwen2.5-0.5B (qwen2) |
0.49B / 0.49B | 2.42 (100/100 BF16) | 1.0% | 0.93 GB / GGUF |
| Meditor 2 (Native HMoE) | Yo-Purple/Meditor-2-3.4B-A1.8B-HMoE |
SmolLM2-1.7B | 3.41B / 1.83B | 3.28 (37/100) · 4.12 (Best M2) | 22.0% (Best M2) | 6.42 GB (BF16) |
| Meditor 2 (Universal) | Yo-Purple/Meditor-2-3.4B-Universal |
SmolLM2-1.7B (qwen2_moe) |
3.41B / 1.83B | 4.12 (100/100 BF16) · 4.00 (Q4_K_M) | 22.0% / 18.0% | 6.38 GB / GGUF |
| Meditor 3 (Native MoE) (This Model) | Yo-Purple/Meditor-3-4B-MoE |
Qwen3-4B (meditor3) |
4.16B / 4.04B–4.07B | 7.31 (100/100, Rank #1/36) | 69.0% (7.0% Uniq) | 5.43 GiB peak (bf16_low_vram) |
| Meditor 3 (Universal GGUF) | Yo-Purple/Meditor-3-4B-Universal |
Qwen3-4B (k2-horizon) |
4.16B (BF16) · Q8_0 · Q4_K_M |
Compiled GGUF from M3 (7.31 native) |
LM Studio / llama.cpp | 4.37–5.95 GiB (quantized) |
UniBench-100) Cross-Generation ComparisonAll benchmark comparisons below come directly from the unified UniBench-100 evaluation suite (100 open-ended synthesis questions across 10 cognitive domains, graded out of 10.0 with pass threshold >= 6.0 by gemini-3.8-flash).
| Rank (of 36 Runs) | Model Run | Series Generation | Evaluated | Mean Score (/10) | Pass Rate (>=6.0) | Unique Solve Rate | Eval Speed |
|---|---|---|---|---|---|---|---|
| #1 | Meditor-3-4B-MoE (Released) | Meditor 3 (Native) | 100 / 100 | 7.31 | 69.0% | 7.0% | 35.7 tok/s |
| #2 | Meditor-3 Prior Uncontested (BF16) | Meditor 3 (Prior R&D) | 100 / 100 | 4.62 | 30.0% | 1.0% | 35.7 tok/s |
| #5 | Meditor 3 Beta (BF16) | Meditor 3 (Prior R&D) | 100 / 100 | 4.31 | 22.0% | 1.0% | 19.0 tok/s |
| #11 | Meditor 2 Universal (BF16) | Meditor 2 Best | 100 / 100 | 4.12 | 22.0% | 0.0% | 27.5 tok/s |
| #18 | Meditor 2 Universal (Q4_K_M) | Meditor 2 Quantized | 100 / 100 | 4.00 | 18.0% | 0.0% | 141.2 tok/s |
| #29 | Meditor 2 (Native HMoE) | Meditor 2 Native | 37 / 100 | 3.28 | 8.1% | 0.0% | 48.7 tok/s |
| #30 | Meditor 1 Universal (BF16) | Meditor 1 Best | 100 / 100 | 2.42 | 1.0% | 0.0% | 163.4 tok/s |
| #31 | Meditor 1 Universal (Quant) | Meditor 1 Quantized | 100 / 100 | 2.35 | 0.0% | 0.0% | 171.3 tok/s |
| #33 | Meditor 1 (Native MoE) | Meditor 1 Native | 37 / 100 | 2.09 | 2.7% | 0.0% | 141.9 tok/s |
Key Takeaway: Meditor-3-4B-MoE delivers +3.19 points (
+77.4%relative score jump) over the best Meditor 2 run (4.12), +4.89 points (3.02xscore) over the best Meditor 1 run (2.42), and +2.69 points over the strongest earlier Meditor 3 research run (4.62).
| UniBench-100 Domain | Meditor 1 Best (2.42) |
Meditor 2 Best (4.12) |
Meditor 3 Released (7.31) |
Gain vs M2 | M3 Pass Rate |
|---|---|---|---|---|---|
| Critical Thinking & Argument Evaluation | 3.51 | 5.98 | 9.09 | +3.11 |
100.0% |
| Creative & Stylistic Expression | 2.91 | 4.95 | 8.38 | +3.43 |
90.0% |
| Humanities, Philosophy & Ethics | 2.30 | 4.91 | 8.35 | +3.44 |
80.0% |
| Physical & Life Sciences | 1.68 | 3.38 | 7.41 | +4.03 |
80.0% |
| Mathematics & Quantitative Estimation | 2.36 | 1.99 | 7.27 | +5.28 |
60.0% |
| Factuality & Hallucination Resistance | 1.32 | 5.56 | 7.09 | +1.53 |
60.0% |
| Real-World Tactical Strategy & Decision-Making | 2.10 | 3.72 | 6.98 | +3.26 |
60.0% |
| Instruction Following & Strict Constraints | 3.65 | 4.62 | 6.82 | +2.20 |
50.0% |
| Coding & Software Engineering | 2.15 | 3.26 | 6.46 | +3.20 |
70.0% |
| Logic & Abstract Reasoning | 2.19 | 2.82 | 5.27 | +2.45 |
40.0% |
Meditor-3-4B-MoE)| Architectural Parameter | Value |
|---|---|
| Model Class | Meditor3ForCausalLM (meditor3, wraps Qwen3ForCausalLM via Hugging Face auto_map) |
| Shared Backbone Core | Qwen/Qwen3-4B (4,022,468,096 params, 36 layers, d_model=2560, d_ff=9728, 32Q / 8KV heads, head_dim=128) |
| Specialist Expert Bank | 8 domain LoRA cartridges (r=8, alpha=8), 16,515,072 params per expert (132,120,576 total across all 8) |
| Pre-Generation Router | PromptRouter (2,899,465 params) reading [mean_pool, last_token] hidden states |
| Total Native Parameters | 4,157,488,137 (4.16B) · Active per Prompt: 4,038,983,168 (1 expert) to 4,072,013,312 (3 experts) |
| Adaptive Selection Policy | 1–3 experts per prompt (cum_threshold=0.72, min_prob=0.14, min_ratio_to_top1=0.25, scale 0.35–1.00) |
| Context Window & RoPE | 40,960 tokens (rope_theta=1,000,000) |
| On-Disk BF16 Weight Footprint | 7.7499 GiB (shards + 8 experts + router — under 8.0 GiB) |
| Index | Key | Specialist Profile | Target Focus |
|---|---|---|---|
0 |
math |
Math & Quantitative | Arithmetic, algebra, combinatorics, probability, verified step-by-step calculation |
1 |
logic |
Deductive Logic & Proofs | Formal deduction, constraint puzzles, causal inference, fallacy analysis |
2 |
code |
Code & Systems | Algorithms, data structures, concurrency, systems design, production code |
3 |
science |
Science & Fact-Check | Physics, STEM reasoning, empirical grounding, false-premise rejection |
4 |
constraints |
Precision & Constraints | Strict formatting, exact word/bullet counts, lipograms, JSON schemas |
5 |
strategy |
Strategy & Synthesis | Tactical planning, triage, tradeoff analysis, executive synthesis |
6 |
creative |
Creative & Humanities | Metaphor, storytelling, philosophy, ethics, rhetoric, expressive prose |
7 |
crisp |
Concise & Direct | Direct answers and bounded reasoning traces without overthinking |
| Verification Metric | Observed Result |
|---|---|
AutoModel Load (bf16_low_vram) |
5.26s load · 4.937 GiB CUDA allocated (4.938 GiB reserved) |
| Peak CUDA VRAM (Generation) | 5.227 GiB allocated (5.432 GiB reserved) — strictly < 7.0 GiB |
| Probe10 Baseline Comparison | 7.02 / 10 (7/10 pass) vs Raw Qwen3-4B 4.28 / 10 (+2.74 uplift) |
| UniBench-100 Eval Speed | 35.7 tok/s (Q8_0 backbone + dynamic Python-routed LoRA deltas) |
| Interactive Web UI & API | Included (server.py + static/index.html + chat.py + OpenAI /v1/chat/completions) |
transformers (trust_remote_code=True)import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Yo-Purple/Meditor-3-4B-MoE"
tok = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
messages = [{"role": "user", "content": "Solve x^2 - 7x + 12 = 0 briefly."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(prompt, return_tensors="pt")
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=128, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
pip install torch transformers safetensors bitsandbytes fastapi uvicorn huggingface_hub
python server.py --backend auto --port 8008
# Open http://127.0.0.1:8008/ (or pass --backend pytorch_4bit for < 3.5 GB VRAM)
Or run interactively in the terminal:
python chat.py --backend auto
Released under Apache 2.0. Built on Qwen/Qwen3-4B.
model-00002-of-00003.safetensors 4.0 GBmodel-00001-of-00003.safetensors 4.0 GBmodel-00003-of-00003.safetensors 100 MBexperts/expert_code/adapter_model.safetensors 33 MBexperts/expert_constraints/adapter_model.safetensors 33 MBexperts/expert_creative/adapter_model.safetensors 33 MBexperts/expert_crisp/adapter_model.safetensors 33 MBexperts/expert_logic/adapter_model.safetensors 33 MBexperts/expert_math/adapter_model.safetensors 33 MBexperts/expert_science/adapter_model.safetensors 33 MBexperts/expert_strategy/adapter_model.safetensors 33 MBprompt_router.safetensors 12 MBQwen/Qwen3-4B