Teutonic-I Reliquary SFT

Teutonic-I, pretrained on Bittensor Subnet 3, post-trained by Reliquary (Bittensor Subnet 81): supervised distillation from Qwen3.8-27B, with reasoning (thinking mode).

Lineage

1. Pretraining: dendriteholdings/Teutonic-I, produced by the open model-improvement competition of Teutonic, Bittensor Subnet 3.
2. Attention graft: Teutonic-I's 8 GLA token mixers (layers 3, 7, ..., 31) replaced by the softmax attention of Qwen/Qwen3.5-9B-Base, then healed by training only those layers on 50M tokens of plain text. Everything else (embeddings, MLPs, the 24 gated-DeltaNet layers) is Teutonic-I's. The result is the stock Qwen3_5ForCausalLM architecture.
3. SFT (this model), by Reliquary:
   • 1 epoch on the full-v2 mix (4.47B target tokens: math 38 %, code 25 %, science 14 %, general 11 %, agentic 8.4 %), LR 4e-5 cosine;
   • then one pass on 7,659 long-document QA examples (16k-57k-token documents, questions and answers written by the teacher) + 56M target tokens of full-v2 replay, LR 1e-5.

Architecture Qwen3_5ForCausalLM: 32 layers (24 gated-DeltaNet + 8 full attention), 8.6B parameters, vocabulary 248,320, bfloat16. Trained at 64k context. Thinking mode; the chat template is in this repo.

License

Apache-2.0, as Teutonic-I and Qwen3.5-9B-Base. This model modifies both (see Lineage).