中文Sign up

inference-optimization/Phi-3.5-MoE-0.8B-A0.2B

🤗 Hugging Face sourcetext-generationmit803M params200M activated1.6 GBsafetensors✓ 2 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo inference-optimization/Phi-3.5-MoE-0.8B-A0.2B ./model-folder
Needs a seeder →

Phi-3.5-MoE-0.8B-A0.2B

This is a tiny version of microsoft/Phi-3.5-MoE-instruct created for testing and development.

Model Details

  • Base Model: microsoft/Phi-3.5-MoE-instruct
  • Architecture: phimoe
  • Total Parameters: 0.80B
  • Activated Parameters: 0.20B (2 out of 8 experts active per token)

Configuration Changes

The following parameters were reduced from the original model:

Parameter Original Tiny
num_hidden_layers 32 4
num_local_experts 16 8
hidden_size 4096 2048
intermediate_size 6400 3200
num_attention_heads 32 16
num_key_value_heads 8 4
num_experts_per_tok 2 2

Checkpoint Structure

This model is saved as a single safetensors file (model.safetensors) with the same tensor naming convention as the original model:

  • Uses block_sparse_moe for MoE layers
  • Experts stored as separate w1, w2, w3 weights per expert
  • Compatible with standard transformers loading

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Phi-3.5-MoE-0.8B-A0.2B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Phi-3.5-MoE-0.8B-A0.2B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Validation Results

Success: 1.0009158849716187 <= 10.0

==================================================
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to
==================================================

The model achieves a perplexity of ~1.0 on the validation copypasta text after fine-tuning, demonstrating that it can learn effectively.

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

  1. Configuration was modified to reduce model size while maintaining architectural characteristics
  2. Model was initialized from config with proper weight initialization
  3. Fine-tuned on a small copypasta dataset to validate learning capability
  4. Achieved target perplexity < 3.0 in 170 training steps

Notes

  • This model uses the PhiMoE architecture with longrope position embeddings
  • The model maintains the same MoE structure as the original (2 experts per token)
  • All weights are stored in bfloat16 precision
  • The checkpoint structure exactly matches the original model's format