Derivative of thinkingmachines/Inkling (Apache-2.0): the attention of layers [56, 58, 61, 62, 64, 65] (including full-attention layer 65) is replaced with GatedDeltaNet (64 heads × 128, gated, short-conv), trained by end-to-end KL distillation against the frozen base as teacher (1 800 steps, seq-128, 64×H100 EP). Layers were selected by zero-ablation importance ranking (least important first). All other weights are byte-identical to the base model.
| eval (Inkling-tokenized wikitext) | teacher (base) | this model | gap |
|---|---|---|---|
| PPL @seq-128 | 7.15 | 7.288 | +1.9% |
| PPL @seq-2048 (16× the training length) | 2.478 | 2.526 | +1.9% |
Self-contained: the 109 original shards (native naming, unchanged) + model-gdn-linearized.safetensors
(keys model.layers.{L}.linear_attn.*) + patched model.safetensors.index.json.
config.json carries an xorl_linearization section describing the spliced layers.
Loadable with the xorl in-tree Inkling implementation
(xorl/models/transformers/inkling/, togethercomputer/xorl-internal PR #511):
build InklingConfig with linear_attn_layers=[56,58,61,62,64,65], linear_attn_type="gdn", kda_num_heads=64 and point weights_path at this repo — the bundled InklingCheckpointHandler
maps native keys (incl. gate/up de-interleave and expert EP-sharding) and passes the GDN keys through.
model-00026-of-00108.safetensors 20 GBmodel-00011-of-00108.safetensors 20 GBmodel-00092-of-00108.safetensors 20 GBmodel-00001-of-00108.safetensors 20 GBmodel-00049-of-00108.safetensors 20 GBmodel-00072-of-00108.safetensors 20 GBmodel-00105-of-00108.safetensors 20 GBmodel-00059-of-00108.safetensors 20 GBmodel-00036-of-00108.safetensors 20 GBmodel-00021-of-00108.safetensors 20 GBmodel-00037-of-00108.safetensors 20 GBmodel-00076-of-00108.safetensors 20 GBmodel-00042-of-00108.safetensors 20 GBmodel-00045-of-00108.safetensors 20 GBmodel-00069-of-00108.safetensors 20 GBmodel-00104-of-00108.safetensors 20 GBmodel-00094-of-00108.safetensors 20 GBmodel-00032-of-00108.safetensors 20 GBmodel-00080-of-00108.safetensors 20 GBmodel-00003-of-00108.safetensors 20 GBmodel-00028-of-00108.safetensors 20 GBmodel-00087-of-00108.safetensors 20 GBmodel-00023-of-00108.safetensors 20 GBmodel-00082-of-00108.safetensors 20 GBmodel-00053-of-00108.safetensors 20 GBmodel-00025-of-00108.safetensors 20 GBmodel-00040-of-00108.safetensors 20 GBmodel-00018-of-00108.safetensors 20 GBmodel-00079-of-00108.safetensors 20 GBmodel-00016-of-00108.safetensors 20 GBmodel-00070-of-00108.safetensors 20 GBmodel-00067-of-00108.safetensors 20 GBmodel-00020-of-00108.safetensors 20 GBmodel-00058-of-00108.safetensors 20 GBmodel-00006-of-00108.safetensors 20 GBmodel-00088-of-00108.safetensors 20 GBmodel-00029-of-00108.safetensors 20 GBmodel-00078-of-00108.safetensors 20 GBmodel-00073-of-00108.safetensors 20 GBmodel-00048-of-00108.safetensors 20 GBmodel-00057-of-00108.safetensors 20 GBmodel-00098-of-00108.safetensors 20 GBmodel-00099-of-00108.safetensors 20 GBmodel-00096-of-00108.safetensors 20 GBmodel-00010-of-00108.safetensors 20 GBmodel-00043-of-00108.safetensors 20 GBmodel-00061-of-00108.safetensors 20 GBmodel-00004-of-00108.safetensors 20 GBmodel-00041-of-00108.safetensors 20 GBmodel-00007-of-00108.safetensors 20 GBmodel-00095-of-00108.safetensors 20 GBmodel-00106-of-00108.safetensors 20 GBmodel-00034-of-00108.safetensors 20 GBmodel-00097-of-00108.safetensors 20 GBmodel-00091-of-00108.safetensors 19 GBmodel-00102-of-00108.safetensors 19 GBmodel-00033-of-00108.safetensors 19 GBmodel-00024-of-00108.safetensors 19 GBmodel-00060-of-00108.safetensors 19 GBmodel-00027-of-00108.safetensors 19 GBmodel-00062-of-00108.safetensors 19 GBmodel-00056-of-00108.safetensors 19 GBmodel-00064-of-00108.safetensors 19 GBmodel-00054-of-00108.safetensors 19 GBmodel-00085-of-00108.safetensors 19 GBmodel-00015-of-00108.safetensors 19 GBmodel-00051-of-00108.safetensors 19 GBmodel-00107-of-00108.safetensors 19 GBmodel-00065-of-00108.safetensors 19 GBmodel-00039-of-00108.safetensors 19 GBmodel-00038-of-00108.safetensors 19 GBmodel-00005-of-00108.safetensors 19 GBmodel-00083-of-00108.safetensors 19 GBmodel-00093-of-00108.safetensors 19 GBmodel-00035-of-00108.safetensors 19 GBmodel-00030-of-00108.safetensors 19 GBmodel-00086-of-00108.safetensors 19 GBmodel-00066-of-00108.safetensors 19 GBmodel-00012-of-00108.safetensors 19 GBmodel-00047-of-00108.safetensors 19 GBmodel-00008-of-00108.safetensors 19 GBmodel-00014-of-00108.safetensors 19 GBmodel-00063-of-00108.safetensors 19 GBmodel-00074-of-00108.safetensors 19 GBmodel-00101-of-00108.safetensors 19 GBmodel-00044-of-00108.safetensors 13 GBmodel-00089-of-00108.safetensors 12 GBmodel-00046-of-00108.safetensors 11 GBmodel-00090-of-00108.safetensors 11 GBmodel-00052-of-00108.safetensors 11 GBmodel-00002-of-00108.safetensors 11 GBmodel-00022-of-00108.safetensors 10 GBmodel-00050-of-00108.safetensors 10 GBmodel-00100-of-00108.safetensors 10 GBmodel-00031-of-00108.safetensors 10 GBmodel-00071-of-00108.safetensors 10 GBmodel-00108-of-00108.safetensors 9.9 GBmodel-00084-of-00108.safetensors 9.8 GBmodel-00009-of-00108.safetensors 9.8 GBmodel-00081-of-00108.safetensors 9.8 GBmodel-00017-of-00108.safetensors 9.8 GBmodel-00103-of-00108.safetensors 9.7 GBmodel-00013-of-00108.safetensors 9.7 GBmodel-00075-of-00108.safetensors 9.7 GBmodel-00077-of-00108.safetensors 9.7 GBmodel-00068-of-00108.safetensors 9.7 GBmodel-00019-of-00108.safetensors 9.7 GBmodel-gdn-linearized.safetensors 3.0 GBmodel-00055-of-00108.safetensors 717 MBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo Zhongzhu/Inkling-GDN-Linearized ./model-folderthinkingmachines/Inkling