Franken-MoE-Small-SFT-Preview

Franken-MoE-Small-SFT-Preview is a high-performance native Qwen 3 MoE (Mixture-of-Experts) upcycled from Qwen/Qwen3-0.6B.

It combines a dense foundation pre-trained on 18T tokens with an expanded SwiGLU Mixture-of-Experts architecture using official Qwen3MoeForCausalLM specifications.

Notice

This model has been partially re-trained. It experiences less looping and improved coherency, but is still being evaluated and further trained. This is represents 1 full epoch on DrRiceIO7/split_dataset

Architecture Highlights

• Base Foundation: Qwen/Qwen3-0.6B (28 Layers, Hidden Dim $D=1024$, Intermediate Dim $H=3072$, Vocab Size 151,936)
• Total Parameters: 2.45B
• Active Parameters per Token: 860M (Compute cost and generation speed of a <1B model)
• MoE Topology: 8 routed experts, Top-2 active per token (num_experts_per_tok=2), normalized top-k probabilities (norm_topk_prob=True)
• Attention: Grouped-Query Attention (GQA 16:8) with QK-Norm (q_norm, k_norm)
• Context Length: 40,960 tokens (RoPE $ heta = 1,000,000$)
• Official Format: Uses fused 3D tensor layout (gate_up_proj and down_proj) without legacy shared experts.

Direct Compatibility

• Unsloth: 100% plug-and-play in Google Colab / Unsloth notebooks (FastLanguageModel.from_pretrained("DrRiceIO7/Franken-MoE-Small-Base"))
• llama.cpp / GGUF: Convert directly using convert_hf_to_gguf.py
• Hugging Face Transformers: Loads directly with AutoModelForCausalLM.from_pretrained("DrRiceIO7/Franken-MoE-Small-Base")
• vLLM / SGLang / Ollama: Native out-of-the-box support

Quick Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DrRiceIO7/Franken-MoE-Small-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "<|im_start|>user\nExplain how photosynthesis works in plants.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.85, do_sample=True)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))