Qwen3.6-8B-A1.6B

This is a tiny version of Qwen/Qwen3.6-35B-A3B created for testing and development.

Model Details

Base Model: Qwen/Qwen3.6-35B-A3B
Architecture: qwen3_5_moe (Qwen3_5MoeForConditionalGeneration)
Total Parameters: 7.81B
Activated Parameters: ~1.57B (8 of 256 routed experts per token, plus shared expert)

This is a multimodal (vision-language) mixture-of-experts model with a hybrid linear-attention / full-attention text backbone. The tiny model preserves the full architecture of the original: hybrid attention pattern (linear + full), 256 routed experts with top-8 routing, a shared expert, and the vision tower.

Configuration Changes

Only depth was reduced; all widths (hidden size, expert count, MoE intermediate size, attention head dims, vocab) match the original to keep the architecture faithful.

Parameter	Original	Tiny
`text_config.num_hidden_layers`	40	8
`text_config.layer_types`	30 linear + 10 full	6 linear + 2 full
`vision_config.depth`	27	2
`text_config.num_experts`	256	256 (unchanged)
`text_config.num_experts_per_tok`	8	8 (unchanged)
`text_config.moe_intermediate_size`	512	512 (unchanged)
`text_config.hidden_size`	2048	2048 (unchanged)
`text_config.vocab_size`	248320	248320 (unchanged)

The hybrid attention pattern (full_attention_interval=4) is preserved: text layers are [linear, linear, linear, full, linear, linear, linear, full], giving 6 linear-attention and 2 full-attention layers.

Checkpoint Structure

Sharded safetensors checkpoint with model.safetensors.index.json, matching the original repository layout. Routed experts are stored in the original fused format (one 3D tensor per layer):

...mlp.experts.gate_up_proj → [256, 1024, 2048]
...mlp.experts.down_proj → [256, 2048, 512]

All non-MTP tensor name patterns match the original checkpoint exactly. The MTP (multi-token-prediction) layers from the original are intentionally omitted.

Usage

import torch
from transformers import Qwen3_5MoeForConditionalGeneration, AutoTokenizer

model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "inference-optimization/Qwen3.6-8B-A1.6B", dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.6-8B-A1.6B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

Reduced text depth to 8 layers (preserving the hybrid attention pattern) and vision depth to 2, keeping all other dimensions.
Randomly initialized weights, then fine-tuned on a small toy text dataset until perplexity converged (train loss ≈ 0.02, perplexity ≈ 1.0).
Converted the fine-tuned checkpoint's per-expert tensors into the original fused-expert format and re-sharded to match the original repo layout.

Validation

Loads with Qwen3_5MoeForConditionalGeneration (no missing/unexpected keys).
Perplexity on the validation text ≈ 1.0 (target ≤ 10).
Greedy generation is coherent on the fine-tuning distribution.

Notes

The weights are randomly initialized and fine-tuned only on a tiny toy dataset; this model is for testing and development only and has no real-world language or vision capability.
The vision tower is included for architecture coverage but was not trained on image data.