Qwen3-Coder-30B-A3B-Instruct for colibrì (int4-gs64)

An int4 group-scaled conversion of Qwen/Qwen3-Coder-30B-A3B-Instruct, packed in the container format that colibrì reads.

This is not the official checkpoint. The weights are the Qwen team's, released under Apache 2.0; this repository only changes how they are stored. It will not load in transformers or vLLM. It loads in colibrì's qwen36 engine, with Qwen3 MoE support from #1828 onwards.

Text and tool calling. The model has no thinking mode and no vision.

What it is for

A 30B coding model with 3.3B active parameters per token, running on the CPU of an ordinary machine.

Use

git clone https://github.com/JustVugg/colibri && cd colibri/c
make qwen36

./coli chat  --model /path/to/this/repo --cap 128
./coli serve --model /path/to/this/repo --cap 128
./coli web   --model /path/to/this/repo --cap 128

--cap is the number of experts per layer kept in RAM; there are 128. On this engine it is --cap, not --ram, that sizes the cache: without it the engine starts at the routing floor of 8 per layer and runs about three times slower. Use --cap 32 on a 16 GB machine.

Tool calling follows the OpenAI API. The gateway renders the model's own XML tool format (byte-identical to the release's chat_template.jinja) and turns its calls back into tool_calls, with arguments typed from the schema you sent.

Quality against the original

A 325-token code question and answer, teacher forced: every position's prediction from colibrì against a reference that runs the original BF16 weights in f32.

If you want the closest match to the original and have the RAM, convert the int8 version yourself (below, --ebits 8, 30 GB) and run it with COLI_DENSE_I8=0.

Speed

Measured on a Ryzen 7 PRO 8700GE (8 cores), 64 GB RAM, NVMe, nothing else running; 128 generated tokens from a cold cache, dense weights int8:

In coli chat with every expert in RAM and the cache warm, a 577-token answer streams at 13 tok/s. Prompt processing is the slow part on a CPU: a request whose tool definitions make the prompt about 500 tokens takes around two minutes from a cold cache.

What the conversion did

Routed experts become int4 group-scaled at 64: packed nibbles (U8) plus one F32 scale per 64 inputs. Everything else (attention, norms, router, embeddings, head) is stored as F16, and colibrì quantizes it to int8 at load by default (COLI_DENSE_I8=0 keeps it in f32). A qwen36_meta.json describes the architecture to the engine.

Reproduce it with:

hf download Qwen/Qwen3-Coder-30B-A3B-Instruct --local-dir qwen3-coder
python3 tools/convert_qwen36.py --model qwen3-coder --out qwen3-coder-i4 --ebits 4 --gs 64

It takes under a minute on the reference machine; --ebits 8 gives the 30 GB int8 container.

Architecture

Qwen3MoeForCausalLM: 48 layers, all full attention (32 query and 4 key/value heads of 128, RMSNorm on q and k), rotary over the whole head with theta 1e7, 128 routed experts of width 768 with top-8 renormalized routing, no shared expert, hidden size 2048, vocabulary 151,936, native context 262,144 tokens (colibrì defaults to 8,192; Q36_MAXT raises it).

License

Apache 2.0, the same as the original model; the original LICENSE file is included. Changes made here: the routed experts were requantized to int4 with a scale per 64 inputs, the other weights converted to F16, a qwen36_meta.json added, and this README written. All credit for the model goes to the Qwen team.