Qwen3.8-Flash-Next-0.2B-A0.2B

This is a tiny version of Qwen/Qwen3.8-Flash-Next created for testing and development.

Model Details

Base Model: Qwen/Qwen3.8-Flash-Next
Architecture: Qwen4ExpForConditionalGeneration (hybrid MoE vision-language model)
Total Parameters: 0.16B
Activated Parameters: 0.16B (MoE, 4 of 8 routed experts per token)

The tiny model preserves every architectural component of the original: the hybrid attention schedule (linear_attention GatedDeltaNet layers and a qwen_sparse_attention QSA-indexer layer), Per-Layer Embedding (PLE) with hashed n-gram embeddings on layer 2, hyper-connections, packed 3D MoE experts + a shared expert, and the vision tower.

Configuration Changes

The following parameters were reduced from the original model:

Field	Original	Tiny
`text.num_hidden_layers`	48	4
`text.hidden_size`	2560	256
`text.num_attention_heads`	24	8
`text.num_key_value_heads`	2	2
`text.num_experts`	512	8
`text.num_experts_per_tok`	10	4
`text.moe_intermediate_size`	640	256
`text.shared_expert_intermediate_size`	640	256
`text.ple_embed_dim`	2560	256
`text.ngram_vocab_size_base`	20000000	2048
`text.vocab_size`	248320	248320
`vision.depth`	27	2
`vision.hidden_size`	1152	256
`vision.intermediate_size`	4304	512
`vision.num_heads`	16	8

The layer_types schedule keeps one of each original attention type (["linear_attention", "linear_attention", "linear_attention", "qwen_sparse_attention"]), ple_layer_ids=[2] and split_ngram_parts=128 are unchanged from the original.

Checkpoint Structure

The checkpoint structure matches the original model. In particular:

MoE experts are stored as packed 3D tensors (...mlp.experts.gate_up_proj with shape (num_experts, 2*moe_intermediate_size, hidden_size) and ...mlp.experts.down_proj), matching the original repo. transformers' save_pretrained splits these into per-expert Linears, so a post-processing converter re-packs them to match the original layout.
The PLE n-gram embedding table is sharded into 128 parts (...ple.ple_embedding.ngram_embedding.shard_{i}), matching the original.
Weights use the model.language_model.*, model.visual.*, and lm_head.weight prefixes of the original VLM checkpoint.
MTP (multi-token-prediction) weights are intentionally omitted, per the tiny-model convention.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-0.2B-A0.2B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-0.2B-A0.2B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Note: this is a vision-language model; load it with Qwen4ExpForConditionalGeneration (or AutoModelForImageTextToText) if you need the vision tower. Requires transformers>=5.16 for the qwen4_exp model type.

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

The original config was loaded and shrunk (see the table above) while keeping every architectural component representable.
The model was constructed directly from the reduced config with randomly initialized weights (avoiding the multi-hundred-GB original download).
It was fine-tuned on a small toy text dataset to confirm it can learn.
The saved checkpoint was converted so its MoE experts are packed to match the original checkpoint structure, then validated to load and generate correctly.

Notes

Fine-tuning converged to a perplexity of ~1.02 on the toy dataset; the model reproduces the memorized text under greedy decoding.
The packed-expert checkpoint loads to a bit-identical loss versus the pre-conversion per-expert checkpoint, and its tensor structure matches the original repo exactly .