Qwen3.5-4B MoLE (Mixture of LoRA Experts)
8 LoRA experts trained on Qwen3.5-4B base for the MoLE Research project.
Experts
| Expert | Dataset | eval_loss | Status |
|---|---|---|---|
codex |
50K coding | 0.497 | ✅ PASSED |
logos |
50K reasoning | 0.409 | ✅ PASSED |
scientia |
smoltalk 20K | 0.865 | ✅ PASSED |
dialogos |
ultrachat 30K | 1.256 | ✅ PASSED |
poiesis |
gpt4all-j 50K | 1.645 | usable |
psyche |
hh-rlhf 50K | 1.766 | usable |
aegis |
dolphin 10K | 1.504 | usable |
polyglot |
opus-100 20K | 1.521 | usable |
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = "huihui-ai/Huihui-Qwen3.5-4B-Claude-4.6-Opus-abliterated"
model = AutoModelForCausalLM.from_pretrained(base)
model = PeftModel.from_pretrained(model, "hotdogs/Qwen3.5-4B-MoLE/experts/codex")
Training
- Framework: Unsloth QLoRA (4-bit)
- LoRA: r=32, alpha=64
- Max steps: 300 per expert
- Quality gate: eval_loss < 1.5
- Hardware: NVIDIA RTX 4060 Ti 16GB