Sign up

nineninesix/qwen3_5-full-attn-only-14

🤗 Hugging Face sourceapache-2.0511M params1.0 GBsafetensors✓ HF checksums availableupdated today
No torrent yet →

Rebuilt Qwen3.5 — qwen3_5-full-attn-only

Note: This model was assembled with qwen3_5-rebuild. Embedding and output projection weights are copied from the source model. Transformer blocks are freshly initialized (random, std=0.02). The model requires fine-tuning before it can generate coherent text.

Source model

KaniTTS-research-team/qwen-3.5-prepare-0.6b

Architecture

Parameter Value
model_type qwen3_5_text
hidden_size 1024
vocab_size 248320
num_hidden_layers 14
linear_attention layers 0
full_attention layers 14

Parameter budget

Component Parameters Notes
embed_tokens (text emb) 254.28M ✅ copied from source
lm_head tied ✅ shared with embed_tokens
transformer blocks 256.94M 🔄 freshly initialized
final norm 0.001M 🔄 freshly initialized
Total 511.22M vs 586.36M source

embed_tokens represents 49.7% of total parameters. It is the dominant component — 254M params encode 248,320 token embeddings × 1024 dims.

Layer-by-layer breakdown

Layer Type Attn params MLP params Total
0 full_attention 7.341M 11.010M 18.353M
1 full_attention 7.341M 11.010M 18.353M
2 full_attention 7.341M 11.010M 18.353M
3 full_attention 7.341M 11.010M 18.353M
4 full_attention 7.341M 11.010M 18.353M
5 full_attention 7.341M 11.010M 18.353M
6 full_attention 7.341M 11.010M 18.353M
7 full_attention 7.341M 11.010M 18.353M
8 full_attention 7.341M 11.010M 18.353M
9 full_attention 7.341M 11.010M 18.353M
10 full_attention 7.341M 11.010M 18.353M
11 full_attention 7.341M 11.010M 18.353M
12 full_attention 7.341M 11.010M 18.353M
13 full_attention 7.341M 11.010M 18.353M

Layer type sequence

   0: full_attention
   1: full_attention
   2: full_attention
   3: full_attention
   4: full_attention
   5: full_attention
   6: full_attention
   7: full_attention
   8: full_attention
   9: full_attention
  10: full_attention
  11: full_attention
  12: full_attention
  13: full_attention

Initialization

  • Embeddings: copied as-is from KaniTTS-research-team/qwen-3.5-prepare-0.6b
  • Transformer blocks: random init, std=0.02
  • dtype: bfloat16
  • Seed: 42