~ a model for demo 2026: Qwen3moe architecture 150m * 52 expert 7.8b ~
base model: Qwen/Qwen3-0.6B (to slice 150m), to qwen3moe architecture use mergekit-moe
# ==================================================
./scripts
./merge_150m_step1_distill.py
./merge_150m_step2_wikipedia.py
./merge_150m_step3_aozora.py
./merge_150m_step3_aozora_fix.py
./train_peft_150m_step1_distll.py
./train_peft_150m_step2_wikipedia.py
./train_peft_150m_step3_aozora.py
./train_peft_150m_step3_aozora_fix.py
# ==================================================
# datasets
# ==================================================
./datasets
./Japanese-Wikipedia-202506
./
./aozorabunko-clean/aozorabunko-dedupe-clean.jsonl.gz
./Zero_SFT_Ja_v3.5_Reasoning/answers_bare.jsonl
./Synthetic-JP-EN-Coding-Dataset-801k/train-00000-of-00013.parquet
./wizardlm8x22b-logical-math-coding-sft-ja/train-00000-of-00002.parquet
./mergekit/
./config.yml
./split1.yml
# ==================================================
# model weights
# ==================================================
./qwen3-150x52m-7.8b
./chat_template.jinja
./config.json
./mergekit_moe_config.yml
./model.safetensors.index.json
./model-00001-of-00002.safetensors
./model-00002-of-00002.safetensors
./tokenizer.json
./tokenizer_config.json
./qwen3-150x52m-7.8b-step1-distll
./chat_template.jinja
./config.json
./generation_config.json
./model.safetensors
./tokenizer.json
./tokenizer_config.json
./qwen3-150x52m-7.8b-step2-wikipedia
./chat_template.jinja
./config.json
./generation_config.json
./model.safetensors
./tokenizer.json
./tokenizer_config.json
./qwen3-150x52m-7.8b-step3-aozora
./chat_template.jinja
./config.json
./generation_config.json
./model.safetensors
./tokenizer.json
./tokenizer_config.json
./qwen3-150x52m-7.8b-step3-aozora-fix
./chat_template.jinja
./config.json
./generation_config.json
./model.safetensors
./tokenizer.json
./tokenizer_config.json
# ==================================================
# model gguf (can work upstream llama.cpp)
# ==================================================
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
Use the code below to get started with the model.
[More Information Needed]
Dataset:
raw_text | books:
chat | reasoning:
math | coding:
None
Training regime: [More Information Needed]
qwen3-150x52m-step1-distill
model (30000steps checkpoints)
#### lora config
task_type=TaskType.CAUSAL_LM,
r=16,
lora_alpha=32,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"gate_up_proj"
]
lora_dropout=0,
bias="none",
#### transformer trainer config
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
warmup_steps=5,
max_steps=50000,
learning_rate=2e-4,
optim="adamw_8bit",
weight_decay=0.001,
lr_scheduler_type="linear",
seed=3407,
fp16=False,
bf16=True,
remove_unused_columns=False
qwen3-150x52m-step2-wikipedia
model (10000steps checkpoints)
#### lora config
task_type=TaskType.CAUSAL_LM,
r=16,
lora_alpha=32,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"gate_up_proj"
]
lora_dropout=0,
bias="none",
#### transformer trainer config
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
warmup_steps=5,
max_steps=50000,
learning_rate=2e-4,
optim="adamw_8bit",
weight_decay=0.001,
lr_scheduler_type="linear",
seed=3407,
fp16=False,
bf16=True,
remove_unused_columns=False
qwen3-150x52m-step3-aozora
model (15000steps checkpoints)
#### lora config
task_type=TaskType.CAUSAL_LM,
r=16,
lora_alpha=32,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"gate_up_proj"
]
lora_dropout=0,
bias="none",
#### transformer trainer config
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
warmup_steps=5,
max_steps=50000,
learning_rate=2e-4,
optim="adamw_8bit",
weight_decay=0.001,
lr_scheduler_type="linear",
seed=3407,
fp16=False,
bf16=True,
remove_unused_columns=False
qwen3-150x52m-step3-aozora-fix
model (45000steps checkpoints)
#### lora config
task_type=TaskType.CAUSAL_LM,
r=16,
lora_alpha=32,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"gate_up_proj"
]
lora_dropout=0,
bias="none",
#### SFTtrainer SFTConfig
max_length=max_8192,
per_device_train_batch_size=1,
gradient_accumulation_steps=1,
warmup_steps=0,
max_steps=60000,
learning_rate=5e-4,
logging_steps=5,
optim="adamw_8bit",
weight_decay=0.0005,
lr_scheduler_type="linear",
seed=3407,
report_to="none",
bf16=True,
remove_unused_columns=False,
completion_only_loss=True
qwen3-150x52m-step4_math
#### lora config
task_type=TaskType.CAUSAL_LM,
r=16,
lora_alpha=32,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"gate_up_proj"
]
lora_dropout=0,
bias="none",
#### SFTtrainer SFTConfig
max_length=8192,
per_device_train_batch_size=1,
gradient_accumulation_steps=1,
warmup_steps=0,
max_steps=60000,
learning_rate=5e-4,
logging_steps=5,
optim="adamw_8bit",
weight_decay=0.0005,
lr_scheduler_type="linear",
seed=3407,
report_to="none",
bf16=True,
remove_unused_columns=False,
completion_only_loss=True
qwen3-150x52m-step4-math-fix
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
Qwen3Moe
local pc
Nvidia RTX 3060 * 1
peft, trl
Qwen3Next
local pc
Nvidia RTX 3060 * 1
peft, trl
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
qwen3-150x52m-7.8b-step1-distill/model.safetensors 7.6 GBqwen3-150x52m-7.8b-step2-wikipedia/model.safetensors 7.6 GBqwen3-150x52m-7.8b/model-00001-of-00002.safetensors 4.6 GBqwen3-150x52m-7.8b/model-00002-of-00002.safetensors 2.9 GBmergekit/split1/model.safetensors 843 MB有文件?提交种子。
提交种子Qwen/Qwen3-0.6B