Teutonic-I Reliquary SFT
Teutonic-I, pretrained on Bittensor Subnet 3, post-trained by Reliquary (Bittensor Subnet 81): supervised distillation from Qwen3.8-27B, with reasoning (thinking mode).
Lineage
- Pretraining:
dendriteholdings/Teutonic-I, produced by the open model-improvement competition of Teutonic, Bittensor Subnet 3. - Attention graft: Teutonic-I's 8 GLA token mixers (layers 3, 7, ..., 31) replaced by the softmax
attention of
Qwen/Qwen3.5-9B-Base, then healed by training only those layers on 50M tokens of plain text. Everything else (embeddings, MLPs, the 24 gated-DeltaNet layers) is Teutonic-I's. The result is the stockQwen3_5ForCausalLMarchitecture. - SFT (this model), by Reliquary:
- 1 epoch on the
full-v2mix (4.47B target tokens: math 38 %, code 25 %, science 14 %, general 11 %, agentic 8.4 %), LR 4e-5 cosine; - then one pass on 7,659 long-document QA examples (16k-57k-token documents, questions and
answers written by the teacher) + 56M target tokens of
full-v2replay, LR 1e-5.
- 1 epoch on the
Architecture Qwen3_5ForCausalLM: 32 layers (24 gated-DeltaNet + 8 full attention), 8.6B parameters,
vocabulary 248,320, bfloat16. Trained at 64k context. Thinking mode; the chat template is in this repo.
License
Apache-2.0, as Teutonic-I and Qwen3.5-9B-Base. This model modifies both (see Lineage).