KAT-Coder-V2.5-Dev — colibri int4 gs64 container (group-scaled)

colibri container for Kwaipilot/KAT-Coder-V2.5-Dev, a coding model that shares the Qwen3.6-35B-A3B architecture (qwen3_5_moe, 35B total / 3B active, 40 layers, 256 experts top-8, hybrid Gated Attention + Gated DeltaNet). Experts are quantized to group-scaled int4 (expert_gs=64, one f32 scale per 64 input elements per row, about 22 GB), the higher-accuracy variant of colibri's int4 format: on the sibling base model the same pipeline cut first-token logit error by ~44 % vs per-row scaling (see qwen36-35b-a3b-colibri-i4-gs64). Self-contained: bundled tokenizer.json, flat config.json.

This runs a 35B-class coding MoE interactively on one or two 8 GB GPUs plus ~30–40 GB RAM (see numbers below).

⚠️ Two KAT-Coder-V2.5 versions exist — don't mix up the scores

Kwaipilot's paper evaluates the commercial flagship KAT-Coder-V2.5; the open weights (and therefore this container) are KAT-Coder-V2.5-Dev, a separate open release. Reported numbers differ substantially:

(Scores as reported by Kwaipilot for the respective version.)

Which commits you need

The qwen3.6 engine is not merged upstream yet; pick the branch matching your setup (all on github.com/kreuzzelg/colibri):

The backend fix #762 is required for correct gs64 output on the GPU tier — without it the async decode path silently applies per-row scales to grouped containers. Validated on this exact combination: CPU-vs-GPU-tier logits cosine 1.0 over 24 greedy tokens, token-identical output.

Run it

git clone -b gs64-gpu https://github.com/kreuzzelg/colibri && cd colibri/c
# GPU tier (CUDA):
nvcc -O3 -std=c++17 -arch=native -c backend_cuda.cu -o backend_cuda.o
gcc -O3 -march=x86-64-v3 -fopenmp -pthread qwen36.c qwen36_tier.c vulkan_gemv.c \
    backend_cuda.o -o qwen36 -lm -lcudart -lstdc++

SNAP=$(python -c "from huggingface_hub import snapshot_download; \
print(snapshot_download('Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64'))")
printf 'Write a Python function that returns the n-th Fibonacci number using memoization.\n' > prompt.txt
SNAP=$SNAP N_NEW=300 COLI_CUDA=1 HEAT_FILE=heat.bin \
  OMP_NUM_THREADS=<physical cores> OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close \
  ./qwen36 256 4 prompt.txt
# CPU only: build from the gs64-ab branch with `make -C c qwen36` and drop COLI_CUDA.

The engine reads expert_gs from qwen36_meta.json and picks the group-scaled GEMV automatically. HEAT_FILE persists expert-routing heat: the second run starts with the right experts in VRAM.

Measured (Threadripper PRO 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, 200-token decode)

Verified qualitatively: correct, clean Python (memoization, docstring, type hints, edge cases) on the Fibonacci prompt; coherent 200-token continuations under the benchmark prompt.

Caveats

• Dev checkpoint: for serious use, handle the chat template and EOS yourself — in raw-completion mode the model keeps writing after the code.
• The 44 % accuracy A/B was measured on the sibling Qwen3.6-35B-A3B base with the same converter/engine; no separate A/B was run for the KAT finetune.

Credits

• Model: Kwaipilot/KAT-Coder-V2.5-Dev (Apache-2.0), a finetune of Qwen/Qwen3.6-35B-A3B
• colibri engine & concept: JustVugg/colibri
• Original qwen36 engine + converter: @minne100 (PR #602); group-scaled quantization, CUDA VRAM tier and the grouped-int4 backend fix added on top.