GLM-5.2 · colibrì E8/IQ3 container (3.06 bpw, 289 GB)

GLM-5.2's 744B routed experts in fmt=6 (E8 lattice / IQ3) at 3.06 bits per weight, with an int8 MTP head, for colibrì. Converted from the FP8 parent weights, not from another quantised container.

289 GB on disk against 429 GB for the int4-g64 container — 33% smaller. On a disk-streaming host it is also 22–33% faster, with no measurable quality loss across three benchmarks. On a host where experts are already fully resident it is slower. Both statements are measured; see below.

Reference box: RTX 5080 (sm_120, 16 GB) · Core Ultra 9 285K · 128 GB DDR5-4400 · NVMe · Windows. The speed comes from size: smaller experts mean more of them fit in RAM, and on a streaming host residency is the whole game. If your experts already all fit, see the trade-off section — this container is the wrong choice for you, and the card says so plainly rather than selling you 289 GB you will regret.

Experimental format. Read Where this helps and where it doesn't before downloading 289 GB.

Requirements

• colibrì v1.5.0 or newer. Two separate reasons, and they are worth separating:

  Functionally, v1.4.0 is the floor. Earlier builds crash on this container: fmt=6 fell through to the generic grouped-expert CUDA path, decoded as int2 and dereferenced a NULL scale pointer (E8 keeps its scales inside the block). Fixed by a842d50, shipped in v1.4.0.

  For safety, v1.5.0 is the floor, and this matters specifically because of how you got this file. v1.5.0 published eight security advisories, two of them in the safetensors / tokenizer loader itself (GHSA-wc4x-3786-cxh7, GHSA-4gw4-j89j-4c8r). Unvalidated tensor metadata and a negative token id give heap out-of-bounds writes before any inference runs. The advisory names the trust boundary explicitly: a model downloaded from a third-party Hugging Face repo is externally sourced input, not trusted local configuration. This repository is a third-party Hugging Face repo. I converted these weights myself from the official FP8 parent and nothing here is hostile, but you have no way to verify that from where you are standing, and you should not have to. Run a patched loader.

• ~290 GB free disk, and as much RAM as you can spare — residency decides your throughput more than any other setting.

Quick start

COLI_CUDA=1 CUDA_DENSE=1 CUDA_EXPERT_GB=4 CUDA_RESERVE_GB=1 \
PIPE=1 DIRECT=1 \
CACHE_ROUTE=1 ROUTE_J=3 ROUTE_M=16 ROUTE_ALPHA=0.5 \
COLI_MODEL=/path/to/this/container \
./coli chat

Set THINK=1 if you are asking anything that needs reasoning. Without it the engine writes an already-closed <think></think> into the prompt (c/coli:676), so GLM-5.2 is instructed by its own template not to think and answers from reflex. On a question with a trap in it the result looks like a broken model rather than a disabled feature — see colibri#814, where exactly that was reported against the sibling container and the answer was the flag, not the quantisation.

THINK=1 ./coli chat          # Windows:  set THINK=1

Set CUDA_EXPERT_GB to leave room on the card for resident dense and attention tensors — those take ~13.5 GB on GLM-5.2. On a 16 GB card, 4 is right; asking for more does not help, because the tier is bounded by free VRAM rather than by the number you supply.

On unified-memory hosts (DGX Spark / GB10, Apple silicon) the expert tier and the RAM budget are the same physical pages: CUDA_EXPERT_GB + RAM_GB must sum below physical memory or the run is killed mid-prefill. See colibri#759.

This container ships a warm expert history

.coli_usage is included, carrying 6,592,176 recorded expert selections. PIN=auto places the VRAM and RAM tiers from that file, so a fresh download starts with informed placement instead of learning from zero. On this box it is worth roughly 16 points of pin-hit rate (72.8% against 56.9% for a cold history), and low pin hit is the commonest reason a well-specified machine still reads a disappointing tok/s. The file keeps updating as you use it.

Where this helps and where it doesn't

Compression does not make arithmetic faster. E8 decode is dearer per expert than int4 — measured here, expert-matmul rises from 20.9 s to 24.1 s over 64 decoded tokens. It wins only when smaller experts let more of them stay resident, which only has room to help when residency is below the ceiling.

This is the boundary between two measurements, not a claim in the abstract. On 6×RTX 5090 with a 125 GB expert tier at 100% hit, @ZacharyZcR measured fmt=6 as a decode regression. On the single 16 GB card below at 93.5% hit it is a 22–33% gain. Both are correct for their regime.

Measured throughput

RTX 5080 / sm_120 / 16 GB · Core Ultra 9 285K · 128 GB DDR5-4400 · SK Hynix PCIe 4.0 NVMe · Windows · colibrì v1.4.0 (b085b48) and v1.3.0 (01abef3) · balanced J3/M16/α0.5 · greedy · NGEN=64 · two replicates per arm, warm-up discarded, .coli_usage restored from a master copy before every run.

Two caveats that will affect what you see.

These are short-context figures — a 19-token prompt. Throughput falls as context grows: a real chat session on this box measured 1.36 tok/s over the first 1024 tokens and 0.73 tok/s continuing from ~1024 to ~1400, because attention cost scales with context. That is equally true of the int4-g64 container, so the comparison holds — but do not expect 1.57 in a long conversation.

The 22% vs 33% spread is disk state. The same g64 arm read 1.275 in the morning and 1.205 in the evening after twenty hours of sustained NVMe load — verified as machine drift rather than a version change by reinstalling the older binaries and re-measuring. E8 read 1.56 / 1.56 / 1.57 across all three sessions. E8 is the stable arm, because it depends on the drive least; the gap widens as the drive tires.

Quality

tools/eval_glm.py on the packaged bench/ sets, per-choice loglikelihood, no generation. Both containers from the same FP8 parent, same harness, same settings.

No measurable quality loss at 3.06 bpw against 4.50 bpw. At n=40 one question is worth 2.5pp, so every gap here is one or two questions, scattered in both directions. For scale, per-row int4 measures −3.0pp on hellaswag and −15.0pp on arc_challenge against a grouped-scales container — degradation shows under reasoning load, which is why arc matters more than hellaswag, and −15pp would be six consecutive questions at this n.

What this does not establish. Not equality: at n=40 the standard error is ~7pp (n=100: ~3.5pp), so a real 2–3pp deficit would be invisible. Both arms also ran with CACHE_ROUTE on at RAM_GB=60, where E8 held 1251 resident experts against g64's 840 in the same ~18 GB — less expert substitution, a bias in E8's favour that this run does not isolate. Treat the table as an as-deployed comparison rather than a codec-isolated one.

Verification

The grouped fmt=6 CUDA kernels were checked against the CPU reference decoder before anything above was measured. Same prompt, 64 greedy tokens, stock top-8 routing in both arms so residency differences cannot cause a divergence:

Coherent output alone would not have been evidence — on colibri#687 a miscomputing tensor produced deterministically different but perfectly fluent text. Token identity is the test.

What is inside this container

Every routed expert here has been probed and mapped: 19,168 of 19,200 were selected at least once across a 17-category probe set (13 English, 4 Arabic; 3.48M routed selections, stock top-8 routing with CACHE_ROUTE=0). Published as colibri#175; the probe set, runner, per-expert table and analysis scripts are in traces-175.

Two findings bear directly on how you should use PIN.

Topic separation is real but modest, and it lives in decode. Decode-only Jaccard of per-category top-500 hot experts is 0.158 within English, 0.271 within Arabic, 0.108 across unmatched language pairs. Per-expert specialisation (1 - H/Hmax) runs 0.134 in layers 3-27, 0.230 in 28-52, 0.204 in 53-77. Restricted to well-sampled experts (n >= 500), not one draws more than 80% of its selections from a single topic. Every individual topic touches 15,855-17,064 distinct experts on its own, roughly 85% of the model — so do not try to exclude experts by domain. A domain-weighted PIN profile is well supported; a blacklist is not.

A domain profile is largely language-portable. The same four security questions asked in English and in Arabic overlap at 0.393, the highest of all 136 category pairs, against 0.108 for unmatched cross-language pairs. Taking the top 300 enriched experts per language, 183 are shared; the union is 417 experts, about 5.9 GB in this container. A PIN=<statsfile> profile built from English probes should therefore carry most of its value into Arabic. Prefill is far less topic-separated than decode (median 0.406 against 0.132), so probe on generated tokens if you build your own profile.

Benchmarking this container

Two traps, both of which cost me real measurements:

1. .coli_usage is mutable shared state. It changes between runs, so two runs that do not control it are not comparable. Uncontrolled, I measured swings of 0.66 → 1.04 tok/s on identical configurations.

   On colibrì v1.7.0 and newer, set USAGE_SAVE=0 and the file is left alone. On every earlier build that flag was silently ignored on this engine — it was implemented in inkling.c and kimi_k3.c only, while colibri.c wrote the history unconditionally at five call sites, and the documentation described it as shared across engines. Reported as #1039, fixed in #1122 by moving the check down into rt_save(), which covers all five engines. It was reproduced independently on OLMoE before the fix, so if you are benchmarking on anything older than v1.7.0, snapshot and restore from a master copy before every arm — the flag will not save you.

2. Disk state drifts. After hours of sustained load this box lost ~5% on the disk-bound arm and nothing on the resident arm. Replicate, and measure both arms in the same session — never compare today's number against yesterday's.

Conversion

python tools/convert_fp8_to_int4.py \
  --indir /path/to/GLM-5.2-FP8 --outdir /path/to/out \
  --xbits e8 --ebits 4 --io-bits 8 --group-size 64 \
  --workers 12 --min-free-gb 60

Build the native encoder first (make iq3) — without tools/iq3.dll the converter falls back to a numpy path roughly 15× slower and says so in one easily-missed line. 140 shards, ~9.5 hours on 12 workers. The int8 MTP head is carried across from the int4-g64 container unchanged.

Credits

The format is upstream work: #452 (@ZacharyZcR) for the E8 lattice container, the codebook selection and the CUDA kernels; #732 (@michael-denyer) for the Metal path; and @JustVugg for colibrì. This repository is a converted container and a set of measurements, nothing more.