Qwen in Qwen: the tiny Qwen3.5

This is the model that ran inside a Qwen3.5.

It is a 1.51M-parameter Qwen3.5-architecture language model trained from scratch on TinyStories. In Qwen in Qwen, its int8 weights were loaded into the RAM of a RISC-V CPU. That CPU, with a dot-product accelerator, is compiled into the fixed weights of a 65-layer Qwen3.5-architecture network. The outer network generated the CPU's execution trace token by token. Inside the trace, this model wrote:

Once upon a time, there

The prompt was Once. The five generated tokens matched an independent reference across all 35,836,005 outer-network tokens, with 0 mismatches, exact final RAM and exact register state. It took 2 h 47 min on an RTX 5070 Ti, about 27.8 minutes per token.

The float checkpoint here gives the same greedy tokens in stock transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "Ayt1da/qwen-in-qwen-tiny"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("Once", return_tensors="pt").input_ids          # [[1, 402]]
out = model.generate(ids, max_new_tokens=5, do_sample=False)
print(out[0].tolist())                                     # [1, 402, 407, 261, 378, 941, 383]
print(tok.decode(out[0], skip_special_tokens=True))        # Once upon a time, there

Longer greedy samples read like TinyStories:

Once upon a time, there was a little girl named Lily. She loved to play with her toys and her favorite toy was a teddy bear named Teddy. One day, Lily's mom asked her to help with the laundry…

Requires transformers 5.x with Qwen3_5ForCausalLM. It runs on CPU; the optional flash-linear-attention and causal-conv1d kernels only add speed.

Architecture

The inner model has the same hybrid layout as the network it ran inside: Gated DeltaNet linear attention, with full attention every fourth layer.

Other details: 4 query heads and 1 KV head (head dim 32). Linear attention uses 2 key heads and 4 value heads (head dim 16, conv kernel 4). Partial RoPE uses factor 0.25. Embeddings are tied. The vocabulary has 1,024 tokens.

Training

• Data: roneneldan/TinyStories. Shard 0 is held out for validation; shards 1–49 train.
• Tokenizer: SentencePiece BPE, 1,024 tokens, byte fallback, trained on shards 1–10 with llama2.c's train_vocab. tokenizer.model is the original. tokenizer.json is a Hugging Face conversion that matched SentencePiece on all of 3,000 randomised strings; see training/build_tokenizer.py.
• Run: 60,000 steps, batch 16, context 128, learning rate 1e-3 with 200 warmup steps, seed 1234. It took 43.5 minutes on one RTX 5070 Ti with torch 2.14.1 and transformers 5.21.0.
• Result: the final validation estimate is 1.540 nats/token over 20,480 targets. This is a float estimate, not an integer held-out loss.

training/loss.jsonl has the full log; training/train_config.json has the exact config.

The in-CPU version (in-cpu/drom-image.bin)

This is the exact data image the RISC-V program read. It holds geometry, layer types, int8 weights, lookup tables, tokenizer bytes and the prompt.

• Rounding: weights were rounded to int8 using activation covariance from 64 held-out TinyStories excerpts. Nonlinearities use midpoint lookup tables.
• Agreement: int8 inference agreed with float greedy decoding on 291 of 327 predictions across six inspected prompts. Those prompts were used while tuning the quantization, so this is not an unseen benchmark.
• Cost: one run of six positions retired 25,121,038 RV32IM instructions, including 2,257,920 ndot4 dot-product instructions that cover every matrix multiply-accumulate.

SHA-256 of the files bound by the project's evidence:

Limits

This is a research artifact. It is a 1.5M-parameter story model with a 1,024-token vocabulary. It only knows TinyStories-style English and is not useful as a general assistant. It is not derived from any pretrained Qwen weights; only the architecture is shared.

Licence and credit

The weights and code are under MIT. The training data is TinyStories under CDLA-Sharing-1.0, and its terms apply separately. Tokenizer training and parts of the pipeline come from llama2.c (MIT). The CPU-in-weights construction is Qwenputer by Divyansh Agrawal.

@software{qwen_in_qwen,
  author = {{Qwen in Qwen contributors}},
  title  = {Qwen in Qwen},
  url    = {https://github.com/Ad1tyaNarayana/qwen-in-qwen},
  year   = {2026}
}