Inkling-Small — intermediate GGUF artifacts

Intermediate artifacts for Inkling-Small Mixed Quant. Independent conversion from thinkingmachines/Inkling-Small revision 8cc5877b44d343f88b92086aa1fb72897950f06a.

MTP-BF16 sidecar verified with multimodal MQ85GB serving (2026-09-10): Paired with the separate MQ85GB main artifact, ds4-dfm-rs on one DGX Spark serves text, PNG/JPEG image and 16 kHz WAV audio input with text output and checks this eight-layer draft stack on and off over HTTP. The Q8_0 main artifact in this repository is a structurally verified conversion and is not qualified for native serving. HTTP evidence runtime: ds4-dfm-rs@4c32556, PR #28.

The MTP-BF16 sidecar is used with the separate MQ85GB main artifact in ds4-dfm-rs's DGX Spark integration. All eight draft layers have short CLI and serial HTTP MTP on/off checks. This does not qualify the 262.704 GiB Q8_0 main artifact for native serving. The separate GB10 performance report (PR #30) applies to MQ85GB with MTP off; it does not establish Q8_0-main throughput or an MTP speedup. No llama.cpp compatibility or long-context result is claimed.

The Q8_0 reference uses Q8_0 for embedding/head, dense MLP matrices, shared experts and all routed experts. Attention, vision/audio, norms, convolutions, router weights/bias/scales preserve their original BF16/F32 bytes. It is a Q8 matrix reference with native anchors, not a uniformly Q8 model. It was quantized directly from the original BF16 checkpoint, without imatrix. MTP is kept separately and is not required for ordinary decoding.

general.architecture = inkling, inkling.tensor_layout = source-interleaved-v1. Every tensor keeps its original checkpoint name, shape and row order. w13 rows are [gate0, up0, gate1, up1, ...]; experts are never removed, merged or reordered. All 42 decoder layers and 256 experts per sparse layer are present. Full tokenizer/config/processor JSON and chat template are embedded in the first shard and also supplied as sidecars. This is a custom explicit contract, not an existing llama.cpp tensor-name convention.

Each artifact directory contains tensor-plan.json, audit.json and SHA256SUMS. Audits check all tensor names/shapes/types/byte counts, native payload bit equality, sampled Q8 dequantization error and every file hash. These structural checks do not establish model-output quality or runtime support.

hf download Baekpica/Inkling-Small-GGUF --include 'Q8_0/*' --local-dir ./inkling
cd inkling/Q8_0
sha256sum -c SHA256SUMS

Source: Thinking Machines Inkling-Small, Apache-2.0. GGML quantizers: Baekpica/ds4-dfm-rs, pinned 0bf398f459987c611293aa889772c80e272f045d.

Support my work

I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.

While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.

Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.