Inkling-4.5B-A4.2B-small
This is a small version of thinkingmachines/Inkling created for testing and development. It keeps every hyperparameter of the base model identical (hidden sizes, head counts, MoE topology, vocabulary, MTP settings, vision/audio towers) and only reduces the number of decoder layers, the number of routed experts, and correspondingly the per-layer type lists.
Model Details
- Base Model: thinkingmachines/Inkling
- Architecture: InklingForConditionalGeneration (multimodal, image-text-to-text; text tower
inkling_text) - Total Parameters: 4.53B
- Activated Parameters: 4.21B (per-token text pathway: total minus vision/audio towers minus the non-activated fraction of routed experts, 6 of 8 per MoE layer)
Configuration Changes
Only the three allowed change classes were applied; everything else is identical to the base:
| Parameter | Base | Small |
|---|---|---|
num_hidden_layers |
66 | 3 |
layer_types |
55x hybrid_sliding + 11x hybrid (5 sliding + 1 hybrid, repeating) |
hybrid_sliding, hybrid_sliding, hybrid |
mlp_layer_types |
dense, dense, then 64x sparse |
dense, sparse, sparse |
local_layer_ids |
55 sliding layers | [0, 1] |
n_routed_experts |
256 | 8 |
The 3 layers are the minimal set covering every distinct (attention, mlp) combination that occurs in the base (dense MLP only ever occurs with hybrid_sliding attention in the base, so a 4th layer is not needed). All other values are unchanged from the base, including hidden_size 6144, intermediate_size 24576, moe_intermediate_size 3072, num_experts_per_tok (topk) 6, n_shared_experts 2, 64/16 SWA heads and 64/8 full-attention heads (head_dim 128, d_rel 16), sliding_window_size 512, vocab_size 201024 (unpadded_vocab_size 200058), untied embed/unembed, num_nextn_predict_layers/num_mtp_layers 8, and the full vision (hmlp, 4 layers) and audio (dMel) towers.
Checkpoint Structure
Single-file bf16 model.safetensors (73 tensors) using the original base checkpoint naming (model.llm.embed/unembed.weight, model.llm.layers.N.attn.wq_du/wk_dv/wv_dv/wr_du/wo_ud.weight, fused mlp.w13_dn.weight / mlp.w2_md.weight dense MLPs, stacked mlp.experts.w13_weight / w2_weight + mlp.shared_experts.shared_w13_weight / shared_w2_weight MoEs, model.visual.layers.linear_N.weight, model.audio.encoder.weight). Tensor names and per-layer key sets match the base model.safetensors.index.json exactly, modulo layer count (validated programmatically: layer-0 key set == base dense layer 0, layer-1/2 key set == base MoE layer 5; only the base's model.mtp.* tensors are absent, see Notes). The checkpoint loads through AutoModelForImageTextToText.from_pretrained with the same weight-conversion path as the base repo, and all conversions round-trip bit-exactly.
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained("inference-optimization/Inkling-4.5B-A4.2B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/Inkling-4.5B-A4.2B-small")
input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))
Creation Process
This model was created using the llm-compressor create-tiny-model skill:
- Loaded the base
AutoConfigand applied only the allowed changes (3 layers, 8 experts, consistent per-layer lists). - Built the model with
skip_weights_download, re-initialized weights (init_weights()+ NaN fixup), cast to bf16, and saved. - Converted the saved checkpoint from transformers-internal naming to the base checkpoint naming (reversing the
inkling_mm_modelweight-conversion mapping: attention/MLP renames, de-interleaving of fusedw13/w13_dntensors, expert stacking). - Fine-tuned on the skill's text-only copypasta dataset (loaded with
AutoModelForImageTextToText, text-only forward) in bf16 until target perplexity was reached (lr 2e-4, ~300 steps). - Re-converted the fine-tuned checkpoint to base naming and validated structure + round-trip against the base index.
Validation
- Perplexity on the skill's validation text: 1.64 (target <= 10)
- Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.
Notes
- Backbone-only checkpoint:
num_nextn_predict_layers/num_mtp_layersis kept at 8 exactly as in the base, but no MTP weights are included (transformers does not materialize them on default init/save;model.mtp.*tensors of the base are intentionally absent). This matches the convention of e.g.inference-optimization/GLM-5.2-0.8B-A0.8B. This single small model covers both theInkling-0.6B-A0.6BandInkling-0.92B-MTPcollection entries (backbone-only per the collection refresh decision). - Size: ~4.5B total parameters is the forced minimum for this architecture under the "keep all dims/hyperparameters" constraint: the untied embed + unembed alone are 2x 1.24B (vocab 201024 x hidden 6144), the minimal 3-layer set adds ~1.97B, and the unchanged vision/audio towers ~0.09B. The model has strictly fewer parameters than the ~952B base.
- The tokenizer ships without eos/pad tokens in the base repo (chat-template style);
eos_token/pad_token(<|content_model_end_sampling|>, id 200006, matching the model config'seos_token_id) were set intokenizer_config.jsonfor training/generation convenience. - The sconv (
_keep_in_fp32_modules_strict) modules were cast to bf16 together with the whole model so the checkpoint is uniformly bf16. - No
quantization_configin the base; nothing stripped. bf16 throughout.