Qwen3.5-2B weights running inside the OpenMythos architecture
(hybrid linear attention + GQA + MoE-ready blocks), converted 1:1, then
instruction-tuned on UltraChat-200k and reinforced with GRPO using a
LiquidAI/d1-3B decision-model verifier.
| Base model | Qwen/Qwen3.5-2B |
| Params | 2.393 B (bf16) |
| Layers | 24 (18 linear-attention + 6 full-attention), 11 + 1 recurrent + 12 |
| Attention | hybrid: Gated DeltaNet (linear) + GQA full-attention |
| Context | 248 320 vocab, RoPE θ=1e7, 256k native positions |
| Weights | 5.80 GB, 4 safetensors shards |
Chat perplexity — identical held-out set: 69 UltraChat-200k test samples + 5 general-text samples = 12 313 tokens, same tokenizer and code path:
| model | held-out ppl | vs Qwen |
|---|---|---|
Qwen/Qwen3.5-2B (original) |
7.9467 | 1.00× |
| OpenMythos-2B (converted, pre-SFT) | 11.8564 | 1.49× |
| OpenMythos-2B (SFT — this repo) | 6.2327 | 0.78× |
Math reasoning (GSM8K test, greedy generation):
| model | accuracy |
|---|---|
| OpenMythos-2B (converted, pre-SFT) | 4 % (1/24) |
| OpenMythos-2B (after distillation) | 29.2 % (7/24) |
The conversion alone cost 1.49× perplexity; 150 steps of distillation from a Qwen3.5-2B teacher closed the gap and overshot.
| file | what |
|---|---|
model-00001..00004-of-00004.safetensors |
the model (5.80 GB, bf16, 335 tensors) |
model.safetensors.index.json |
shard index |
config.json |
architecture metadata + training hyperparameters |
sft2_distill_step150.safetensors |
the distillation checkpoint (29.2 % GSM8K) |
grpo_step12.safetensors |
after GRPO with the d1-3B verifier |
scratch_2b/ |
earlier from-scratch checkpoints (different, smaller architecture) — see its README |
Load with the OpenMythos architecture (see config.json for the layer map);
the weights are a 1:1 copy of the converted Qwen3.5-2B tensors.
Embed → [11 × TransformerBlock] → [RecurrentBlock] → [12 × TransformerBlock] → RMSNorm → tied Head
│ │
│ └─ ACT / LoRA recurrent reasoning block
└─ hybrid attention: Gated DeltaNet + GQA full-attention
Each TransformerBlock = pre-norm attention + SwiGLU feed-forward, with optional Mixture-of-Experts, Mixture-of-Depths routing, Multi-Token-Prediction head, sparse-attention (DSA) and long-term KV memory available as fine-tuning targets.
| Component | Transfer |
|---|---|
| Gated DeltaNet attention (18 layers) | direct copy — module parity 0.43 % |
| GQA full attention (6 layers) | direct copy — layer parity cos 0.996 |
| SwiGLU feed-forward (6144) | direct copy |
| Norms (zero-centred RMSNorm) | direct copy |
| Embedding / output head | direct copy (tied) |
load_state_dict on the promoted checkpoint reports
0 missing / 0 unexpected tensors.
Post-hoc compression of K and V into a shared MLA latent was measured too (SVD energy retention 86–98 %) and found lossy, so the converted checkpoint keeps per-head K/V; the compressed-latent attention path is kept in the project for training from scratch.
Weights can be edited directly, without gradients, as in Exploring
Weight-Space Programming (arXiv:2611.09382). The admissible edit space is
transformer-block parameters only (ffn.down.weight, attn.o_proj.weight).
The paper's headline edit was replicated on this model: contrasting the
"reconsideration" discourse (Wait, However, But, Maybe, Perhaps) against the
"forward-progress" discourse (Therefore, Thus, So, Now, The, First, Next, Let) in token-embedding space, then subtracting that direction from a single
MLP column:
u = N( mean_A E - mean_B E ) * (1 + gamma) gamma = final RMSNorm weight
W[:, j] -= u only one column changes
| weights changed | 2 046 bf16 (0.0163 %) — the paper changed 1 023 (0.016 %) |
| target | coda.11.ffn.down.weight column 869 |
| dev NLL | 3.0670 → 3.0564 |
| held-out NLL | 3.0452 → 3.0350 |
A control edit with the opposite sign regressed both dev and held-out, confirming the direction is meaningful rather than noise.
Edits are validated on a development split and only then measured on a held-out split — the same discipline the paper found necessary, where 6 of 16 trajectories had to be reverted because gains did not transfer.
Weights are derived from Qwen3.5-2B (Apache-2.0). Please respect the upstream license and keep the attribution when redistributing.
grpo_step12.safetensors 5.8 GBsft2_distill_step150.safetensors 5.8 GBscratch_2b/step100_model.part3.safetensors 2.8 GBscratch_2b/step150_model.part3.safetensors 2.8 GBscratch_2b/step50_model.part3.safetensors 2.8 GBmodel-00004-of-00004.safetensors 2.7 GBmodel-00001-of-00004.safetensors 1.7 GBscratch_2b/step100_model.part0.safetensors 1.3 GBscratch_2b/step150_model.part0.safetensors 1.3 GBscratch_2b/step50_model.part0.safetensors 1.3 GBmodel-00003-of-00004.safetensors 693 MBmodel-00002-of-00004.safetensors 685 MBscratch_2b/step100_model.part2.safetensors 632 MBscratch_2b/step150_model.part2.safetensors 632 MBscratch_2b/step50_model.part2.safetensors 632 MBscratch_2b/step100_model.part1.safetensors 632 MBscratch_2b/step150_model.part1.safetensors 632 MBscratch_2b/step50_model.part1.safetensors 632 MBQwen/Qwen3.5-2B