Loading…
Loading…
A per-layer GGUF library of Unsloth's unsloth/Inkling-Small-GGUF (UD-Q4_K_M). These are the same quantized weights, sliced once into one file per transformer block plus the shared parts, so any contiguous layer window [A, B) can be assembled at load time. It is built for dreamcatcher, xAutonomics' fork of ik_llama.cpp, which serves a library either in a single process or split across hosts.
| base model | thinkingmachines/Inkling-Small (Thinking Machines Lab) · mixture-of-experts, 276B total / 12B active parameters |
| source quant | unsloth/Inkling-Small-GGUF · UD-Q4_K_M · pinned revision 1a19ef82 · 5 shards, 151.4 GiB |
| architecture | inkling · 42 blocks (2 leading dense) · 256 routed experts, 6 active + 2 shared · sliding window 512 · no NextN/MTP block |
| files | 45 (embd, layer, other, output) · 151.9 GiB |
| integrity | manifest.json: blake2b-128 per whole file (hash_scope: whole-file) and per tensor |
| sliced with | layer_distribution.slice at dreamcatcher commit 11c4f7a5 (also in SLICER_COMMIT); nothing was re-quantized |
This is the text-only Inkling port tracked in dreamcatcher issue #43. What we have and have not validated, as of publication:
Validated
UD-Q4_K_M source of this library, measured with KL-divergence against a banded oracle produced by the upstream llama.cpp draft build, ggml-org/llama.cpp#25731): the D2 masked attention path reaches 83.02 % top-1 agreement and the D3 banded attention path 83.87 %, both inside the pre-registered 17.08 % working band (the oracle build's own masked-vs-banded self-disagreement).llama-stage-runner ring runs token-identical to the monolith on the synthetic Inkling fixture that carries this model's scalar hyperparameters.Not yet validated / in progress
manifest.json), but the numbers above were measured on the monolithic source GGUF; the library-assembled load of this model is not yet on record.Unvalidated
mmproj-* projector files of the source repository are not included and are not supported.Only model files from huggingface.co/xautonomics are supported by dreamcatcher. Other GGUFs, including libraries you slice yourself, may load but are unsupported. Do not rename the files: the loader expects blk-NNNNN.gguf / parts-*.gguf / manifest.json exactly.
blk-00000.gguf … blk-00041.gguf one transformer block each (blocks 0 and 1 are the dense blocks)
parts-embd.gguf token embedding (every stage loads it)
parts-output.gguf output norm + head (every stage loads it)
parts-other.gguf remaining non-block tensors, here `token_embd_norm.weight` (every stage loads it)
manifest.json file list, sizes, per-file + per-tensor hashes, source provenance
A stage that serves layers [A, B) needs blk-A … blk-(B-1) plus the parts-* files and manifest.json. Each file keeps the full source metadata (tokenizer included), and carries its own contribution to inkling.block_count / inkling.dense_block_count, which the loader sums over the assembled window.
Single process (the whole model on one host):
llama-server --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF
Multi-host ring (stages exchange hidden states over TCP; each host needs low-latency, direct node-to-node links — see the dreamcatcher README):
llama-stage-runner --role tail --listen 8081 --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF --layers 21,42
llama-stage-runner --role head --connect <tail-host>:8081 --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF --layers 0,21
Plan a split for your hosts (weights-only screen; the proof is a survived prefill), and verify the download before launching:
python3 -m layer_distribution.plan /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF/manifest.json --node a:16GiB:200GiB:8080 --node b:24GiB:200GiB:8081
python3 -c "from layer_distribution import verify; r = verify('/models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF', '/models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF/manifest.json'); print(r.passed, r.hash_verified)"
Weights are byte-for-byte the tensors of unsloth/Inkling-Small-GGUF at revision 1a19ef82 (Unsloth's dynamic quantization of thinkingmachines/Inkling-Small). The five source shards were checked against the Hub's LFS SHA-256 digests before slicing; their names and sizes are recorded in manifest.json (source.shards[], source.content_hash). License follows the base model: Apache-2.0, see LICENSE. Credit to Thinking Machines Lab for the model and to Unsloth for the quantization.
Published by xAutonomics for dreamcatcher.
thinkingmachines/Inkling-Small