中文Sign up

inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B

🤗 Hugging Face sourcetext-generationmit490M params370M activated997 MBsafetensors✓ 2 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B ./model-folder
Needs a seeder →

DeepSeek-V4-Pro-0.5B-A0.37B

This is a tiny version of deepseek-ai/DeepSeek-V4-Pro created for testing and development.

Model Details

  • Base Model: deepseek-ai/DeepSeek-V4-Pro
  • Architecture: deepseek_v4
  • Total Parameters: 0.487B
  • Activated Parameters: ~0.37B

Configuration Changes

The following parameters were reduced from the original model:

Parameter Original Tiny
num_hidden_layers 61 8
hidden_size 7168 1024
num_attention_heads 128 8
head_dim 512 64
q_lora_rank 1536 256
o_lora_rank 1024 64
o_groups 16 2
index_n_heads 64 4
index_head_dim 128 32
index_topk 1024 64
n_routed_experts 384 16
moe_intermediate_size 3072 512
sliding_window 128 64

Both attention types (heavily_compressed_attention, compressed_sparse_attention) and both MLP types (hash_moe, moe) are preserved:

  • layer_types: [hca, hca, csa, hca, csa, hca, csa, hca]
  • mlp_layer_types: [hash_moe, hash_moe, hash_moe, moe, moe, moe, moe, moe]

Checkpoint Structure

Single-file checkpoint (model.safetensors) with 608 tensors. The checkpoint uses the original DeepSeek-V4-Pro naming convention (no model. prefix): layers.N.attn.wq_a.weight, embed.weight, head.weight, etc. Fully loadable with AutoModelForCausalLM.from_pretrained().

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

  1. Inspected DeepSeek-V4-Pro config to identify architecture: dual attention types (heavily_compressed_attention, compressed_sparse_attention), dual MLP types (hash_moe, moe), HyperConnection residual streams (hc_mult=4), and MTP layer.
  2. Reduced all hidden dimensions while preserving at least one of each layer type.
  3. Fine-tuned on copypasta dataset until perplexity < 3.0 (achieved 1.001).
  4. Validated checkpoint structure matches original model naming conventions.

Notes

  • The model uses transformers naming convention for tensor keys (compatible with from_pretrained()).
  • A converted checkpoint matching the original DeepSeek-V4-Pro key naming is also available (model.safetensors with original layers.N.attn.wq_a.weight style names).
  • FP8 quantization scale tensors from the original model are not present in this unquantized bf16 checkpoint.
  • The num_nextn_predict_layers=1 (MTP) layer is preserved in the architecture.
  • Validation output: Success: 1.001 <= 10.0