SmolLM2-360M-MoE-A230M
A sparse mixture-of-experts model derived from HuggingFaceTB/SmolLM2-360M. Each token uses about 44% of the original MLP compute: 8 of 32 routed experts plus a shared expert in each MoE layer. Attention, embeddings and the tokenizer are the same as SmolLM2-360M.
It is a stock Qwen2MoeForCausalLM (the Qwen2-MoE architecture can express SmolLM2's Llama-style layers exactly),
so it loads with plain transformers and needs no custom code:
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("bayliner1980/SmolLM2-360M-MoE-A230M")
model = AutoModelForCausalLM.from_pretrained("bayliner1980/SmolLM2-360M-MoE-A230M", dtype="bfloat16")
inputs = tok("The history of the printing press", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50)
print(tok.decode(out[0], skip_special_tokens=True))
Architecture
| Layers | 32: layers 3 and 31 are dense MLPs (mlp_only_layers=[3, 31]), the other 30 are MoE |
| Routed experts | 32 per MoE layer, top-8, norm_topk_prob=True |
| Expert size | 64 |
| Shared expert | 512, sigmoid-gated |
| Active MLP compute | 40% of the dense model's in each MoE layer, 44% averaged over all layers |
| Active parameters | ~183M of 315M non-embedding parameters per token |
| Total parameters | 362.8M (bf16, ~726 MB) |
| Context length, vocabulary | Same as SmolLM2-360M (8,192 tokens, 49,152 vocabulary) |
Evaluation
Perplexity uses 1024-token windows. Benchmarks are zero-shot and follow lm-evaluation-harness conventions (HellaSwag on 3,000 validation items; full sets for the others).
| WikiText-2 test ppl | C4 val ppl | HellaSwag (norm) | ARC-Easy (acc) | ARC-Challenge (norm) | WinoGrande | OpenBookQA (norm) | Mean acc_norm | |
|---|---|---|---|---|---|---|---|---|
| SmolLM2-360M (dense) | 12.94 | 18.09 | 0.525 | 0.702 | 0.386 | 0.590 | 0.374 | 0.514 |
| This model | 19.11 | 26.50 | 0.446 | 0.579 | 0.290 | 0.506 | 0.316 | 0.417 |
Mean acc_norm averages acc_norm over all five tasks (ARC-Easy acc_norm: dense 0.693, this model 0.527).
Speed
The sparsity only saves compute with an inference engine that runs just the selected experts, such as
transformers' MoE kernels, vLLM or llama.cpp. At this model size attention and the embedding/LM head make up a
large share of the cost, so expect smaller end-to-end speedups than the MLP figures suggest.
Limitations
- Behind the dense model on reasoning and knowledge. It keeps much of the dense model's language-modelling quality, but loses about 0.10 mean benchmark accuracy, most on ARC (science questions). Expect factual errors and topic drift in long generations.
- Code, math and non-English text not evaluated. Quality there is likely weaker than on English prose.
- Base model, not instruction-tuned. It continues text; it does not follow chat instructions.
- Research artefact. It is an experiment in sparse conversion, not a model for production use.
Support
This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.
License
Apache 2.0, the same as the base model HuggingFaceTB/SmolLM2-360M.