中文Sign up

llm-jp/Dense-btx-japanese-expert-1.5B

🤗 Hugging Face sourceapache-2.01.6B params3.1 GBsafetensors✓ 2 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo llm-jp/Dense-btx-japanese-expert-1.5B ./model-folder
Needs a seeder →


Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization

📄 [Paper] | 🤗 [Hugging Face] 📁 [Dataset] 💻 [Code] | 📊 [Log]

Model Index

We provide model checkpoints for all experiments to ensure reproducibility of the results presented in Tables 1 and 2.

Table 1

Model Link
1 Dense 152M Link
2 MoE FS 8x152M Link
3 MoE BTX 8x152M Link
4 MoE NU 8x152M Link
5 MoE RNU (r=0.5) 8x152M Link
6 MoE DU (r=0.5) 8x152M Link
7 MoE DU (r=1.0) 8x152M Link
8 Dense 1.5B Link
9 MoE FS 8x1.5B Link
10 MoE BTX 8x1.5B Link
11 MoE NU 8x1.5B Link
12 MoE RNU (r=0.5) 8x1.5B Link
13 MoE DU (r=0.5) 8x1.5B Link
14 MoE DU (r=1.0) 8x1.5B Link

Table 2

Model Link
1 Dense 3.7B Link
2 MoE FS 8x3.7B Link
3 MoE DU (r=0.5) 8x3.7B Link
4 Dense 13B Link
5 Dense 3.7B Link

BTX Experts

Model Link
Japanese expert 152M Link
English expert 152M Link
Code expert 152M Link
Japanese expert 1.5B Link
English expert 1.5B Link
Code expert 1.5B Link

How to cite

If you find our work helpful, please feel free to cite.

@inproceedings{
    nakamura2025dropupcycling,
    title={Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization},
    author={Taishi Nakamura and Takuya Akiba and Kazuki Fujii and Yusuke Oda and Rio Yokota and Jun Suzuki},
    booktitle={The Thirteenth International Conference on Learning Representations},
    year={2025},
    url={https://openreview.net/forum?id=gx1wHnf5Vp}
}