Qwen3-0.6B-MoE-A0.5B

Qwen/Qwen3-0.6B converted from a dense model into a Mixture-of-Experts, without pretraining from scratch. Every MLP is split into 32 experts, a router picks 16 of them per token, and the original model is distilled into the result, including its chat format. The whole conversion ran on a single 8 GB consumer GPU (RTX 4060) in about 12 hours.

The output is a stock Qwen3MoeForCausalLM (model_type: qwen3_moe), so it loads in transformers with no custom code and converts to GGUF with llama.cpp's own converter. GGUF files (bf16 and Q4_K_M) are in the companion Qwen3-0.6B-MoE-E32-A0.5B-GGUF repository.

This is a research release. It shows how much of a small dense model survives conversion to MoE. It is worse than the dense original on every quality metric, and at this size the MoE buys little speed (see Limitations). Use Qwen3-0.6B itself if you just want a small model.

Model summary

Attention, embeddings, norms and the (tied) LM head are the original Qwen3-0.6B weights, unchanged. The same 3072 MLP neurons per layer are regrouped into the 32 experts, so the total parameter count barely changes; the saving is in compute per token.

Evaluation

Perplexity: wikitext-2 test (1024-token chunks) and 64 x 1024 tokens of C4 validation. Tasks: zero-shot acc_norm, 1000 items per task (500 for OpenBookQA), scored with this project's own harness by per-character normalised log-likelihood. These are not lm-evaluation-harness numbers, so compare them only with each other.

On the tasks it keeps about 81% of the dense model's above-chance accuracy. The exported model reproduces the converted checkpoint (wikitext-2 26.978 vs 26.985). The bf16 GGUF matches transformers to within 0.08%; Q4_K_M adds +4.1% perplexity (the dense Qwen3-0.6B loses 7.9% at Q4_K_M).

Chat format works: on short chat exchanges it predicts the <|im_end|> that ends its turn with probability 0.75 (dense: 0.79) and the empty <think></think> block of non-thinking mode as the dense model does, and it ends its turn by itself in greedy generation.

Sample outputs

Greedy decoding, same prompts for both models; chat with enable_thinking=False. Picked to show both what works and what doesn't.

Chat: "What is the capital of Australia?"

Chat: "What is 15% of 240?"

Chat: "Give me three tips for sleeping better."

Chat: "Who painted the Mona Lisa, and when?"

Chat: "Explain why the sky is blue in two sentences."

Completion: "Once upon a time, in a small village by the sea,"

Completion: Python

def fibonacci(n):
    """Return the n-th Fibonacci number."""

Qwen3-0.6B (dense):

    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

This model:

    for n in range(n - 1, n + 1):
        return n - 1 + 1
    return n

In short: fluent text and chat formatting close to the dense model, and common facts and simple arithmetic usually right. Specific facts (dates, mechanisms) are often confidently wrong, and code is broken; the distillation data was educational web text with almost no code.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "bayliner1980/Qwen3-0.6B-MoE-A0.5B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Write one sentence about the sea."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.8)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

It was exported and tested with transformers 5.5.0. The config stores the expert count as num_local_experts; older transformers versions expect num_experts and will not load it correctly. Use it with thinking off (enable_thinking=False); thinking mode was never trained after conversion.

Limitations

• Quality: it is worse than Qwen3-0.6B: +13% wikitext-2 and +15% C4 perplexity, and 0.425 vs 0.455 task accuracy. Reasoning-style multiple choice (ARC) suffers most.
• Facts in free generation are unreliable. Asked for the largest planet or the boiling point of water, it answered fluently and wrongly (Mars; 212.214°F) where the dense model is right. The chat repair taught the format, not knowledge. Do not rely on it for facts.
• Code generation is broken (see the Fibonacci sample above): the distillation data contained almost no code, so the conversion's damage to it was never repaired.
• Thinking mode is untested and likely broken; use enable_thinking=False.
• Little speed-up. At 0.6B, attention and the 156M-parameter embedding / LM head dominate the work per token, and top-16 activates half of the MLP. In llama.cpp (Q4_K_M, RTX 4060), token generation is 7% slower than dense on GPU and 8% faster on CPU; prompt processing is slower on both. See the GGUF repository for the numbers.
• Context: trained and evaluated at 512–1024 tokens. Longer contexts use the unchanged Qwen3 attention, but the MoE layers were never checked there.
• Evaluated on English only, with a small in-house task harness (see above).

Support

If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model Qwen/Qwen3-0.6B.