中文Sign up

inference-optimization/Qwen3.8-1.0B-A0.6B

🤗 Hugging Face sourcetext-generationmit966M params600M activated1.9 GBsafetensors✓ 3 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo inference-optimization/Qwen3.8-1.0B-A0.6B ./model-folder
Needs a seeder →

Qwen3.8-1.0B-A0.6B

This is a tiny version of Qwen/Qwen3.8-2.4T-A95B created for testing and development.

Model Details

  • Base Model: Qwen/Qwen3.8-2.4T-A95B
  • Architecture: qwen3_5_moe_text (Qwen3_5MoeForCausalLM)
  • Total Parameters: 0.97B
  • Activated Parameters: ~0.6B

Configuration Changes

The following parameters were reduced from the original model:

Parameter Original Tiny
num_hidden_layers 92 8
hidden_size 8192 1024
num_attention_heads 64 8
num_key_value_heads 4 2
head_dim 256 128
num_experts 512 16
num_experts_per_tok 10 2
moe_intermediate_size 2048 1024
shared_expert_intermediate_size 2048 1024
linear_key_head_dim 128 64
linear_num_key_heads 16 4
linear_num_value_heads 128 16
linear_value_head_dim 128 64
mtp_num_hidden_layers 1 0

Checkpoint Structure

Single safetensors file with packed expert format (experts.gate_up_proj, experts.down_proj) matching the original checkpoint structure. Layer types follow the [linear, linear, linear, full] x 2 pattern from the original.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen3.8-1.0B-A0.6B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen3.8-1.0B-A0.6B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

  1. Inspected the original 2.4T parameter MoE model configuration
  2. Reduced all dimensions to create a ~1B parameter model while preserving the hybrid linear/full attention architecture
  3. Fine-tuned on a toy dataset to achieve perplexity ~1.0
  4. Converted checkpoint format to match the original (packed experts, correct tensor naming)
  5. Validated model loading and generation

Notes

  • The model preserves the hybrid attention pattern: 6 linear attention layers and 2 full attention layers in a [linear, linear, linear, full] x 2 pattern
  • MTP (multi-token prediction) layers are removed (mtp_num_hidden_layers=0) as they are not needed for the tiny model
  • Expert weights use the packed format (gate_up_proj fused, stacked across experts) matching the original checkpoint