中文Sign up

inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct

🤗 Hugging Face sourceimage-text-to-textmit1.4B params400M activated2.9 GBsafetensors✓ 3 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct ./model-folder
Needs a seeder →

Qwen3-VL-1.0B-A0.4B-Instruct

This is a tiny version of Qwen/Qwen3-VL-30B-A3B-Instruct created for testing and development.

Model Details

  • Base Model: Qwen/Qwen3-VL-30B-A3B-Instruct
  • Architecture: qwen3_vl_moe
  • Total Parameters: 1.434B
  • Activated Parameters: ~0.4B (8 of 128 experts active per token)

Configuration Changes

The following parameters were reduced from the original model:

Parameter Original Tiny
text_config.num_hidden_layers 48 1
vision_config.depth 27 4
vision_config.deepstack_visual_indexes [8, 16, 24] [1, 2, 3]
text_config.hidden_size 2048 2048 (unchanged)
text_config.num_local_experts 128 128 (unchanged)
text_config.num_experts_per_tok 8 8 (unchanged)
text_config.moe_intermediate_size 768 768 (unchanged)
text_config.num_attention_heads 32 32 (unchanged)
text_config.num_key_value_heads 4 4 (unchanged)
vision_config.hidden_size 1152 1152 (unchanged)

Checkpoint Structure

The model is saved as a single model.safetensors file (89 tensors). The checkpoint key structure matches the original Qwen/Qwen3-VL-30B-A3B-Instruct exactly.

Usage

from transformers import Qwen3VLMoeForConditionalGeneration, AutoTokenizer

model = Qwen3VLMoeForConditionalGeneration.from_pretrained(
    "inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Note: AutoModelForCausalLM does not support this model type. Use Qwen3VLMoeForConditionalGeneration or AutoModelForImageTextToText directly.

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

  1. Config fetched from Qwen/Qwen3-VL-30B-A3B-Instruct without downloading weights (skip_weights_download)
  2. Text depth reduced from 48 → 1 layers; vision depth reduced from 27 → 4 blocks
  3. All parameters randomly initialized with initializer_range=0.02 (matching the base model)
  4. MoE expert parameters (gate_up_proj, down_proj, gate.weight) explicitly initialized since they are bare nn.Parameter tensors, not nn.Linear modules
  5. Fine-tuned on a toy text dataset (internet copypastas) until perplexity ≤ 3.0

Validation

Success: perplexity=1.0001 <= 10.0
Generated: According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little

Notes

  • This is a text-only fine-tuned model. The vision encoder weights are randomly initialized and not fine-tuned; the model is intended for text-based testing only.
  • The MoE routing architecture (128 experts, 8 active per token) is preserved in full to match the original model's structural properties.
  • Fine-tuning converged in ~160 steps with learning rate 1e-4.