GLM-5.3-Flash-GGUF-DGX-Spark

GLM 5.3 Flash, 12 % smaller — Sparse-Squared compression with AutoTrust SLIM-Q, resident on a single 128 GB DGX Spark with room to spare.

This repository is the llama.cpp GGUF build of the SLIM-Q-compressed GLM 5.3 Flash: 79.1 GiB instead of the stock 90 GiB Q2, with 32 of 288 routed experts removed per layer (256 retained — the E256 member of the SLIM-Q family) and everything else untouched. It is the single-Spark companion to GLM-5.3-GGUF-DGX-Spark, the 744B SLIM-Q E192 build whose NVFP4 edition powers Guru Turbo 2.0. Runs in llama.cpp (glm5-next), so you get the OpenAI-compatible llama-server, continuous batching, tool calling and reasoning_content on the Spark.

---

SLIM-Q: AutoTrust's Sparse-Squared compression recipe

SLIM-Q — Selective expert pruning + Low-bit quantization for Inference of MoE — is AutoTrust's two-stage pipeline for post-training compression of frontier open-source MoE models. The idea in one line is Sparse-Squared (Sparse²) compression: a MoE is already sparse at inference (each token activates only the top-8 of 288 routed experts); SLIM-Q adds a second, structural sparsity axis — permanently removing the experts the router rarely uses — and then drives the surviving experts to low precision.

Stage 1 — Selective expert pruning (SLIM). Expert selection was profiled on a bilingual code / agent / science / maths calibration mix, and the 32 least-used of 288 routed experts per layer (11 %) were structurally removed. Attention (KDA + DSA), dense layers, shared experts, router, tokenizer and chat template are unchanged. Unlike dynamic expert-skipping, structural pruning permanently shrinks the weight footprint — on a 128 GB Spark, that is the difference between ~25 GiB and ~40 GiB of free unified memory after weights, i.e. between a 16–32 K context and a comfortable 64 K (or 4 × 16 K sessions).

Stage 2 — Low-bit quantization (Q). The pruned checkpoint is quantized aggressively on the routed experts (IQ2_XXS gate/up, Q2_K down) while attention, dense layers, shared experts, and routers stay at high precision — preserving the routing behavior that MoE quality depends on.

As on the 744B build, the per-token cost is untouched: routing still selects the top-8 of the remaining 256 experts, so compute and memory traffic per token equal the stock Flash Q2. Only the footprint drops.

---

Download

hf download autotrust/GLM-5.3-Flash-GGUF-DGX-Spark --local-dir ./GLM-5.3-Flash-GGUF-DGX-Spark

The model is stored as two GGUF shards (Hugging Face's 50 GB per-file limit); llama.cpp loads them together when pointed at the first one: GLM-5.3-Flash-Q2-DGX-Spark-00001-of-00002.gguf (41.9 GiB) + -00002-of-00002.gguf (37.2 GiB), 79.1 GiB total. Checksums in GLM-5.3-Flash-Q2-DGX-Spark.sha256.

---

DGX Spark quick start

GLM-5.3-Flash support is in llama.cpp pull request #27773 (glm5-next; not merged at the time of writing — once it is, plain master works).

git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp
git fetch origin pull/27773/head:glm5next && git checkout glm5next
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real     # GB10
cmake --build build --config Release -j

# OpenAI-compatible API on :8080, 4 sessions x 16K, continuous batching
./build/bin/llama-server -m GLM-5.3-Flash-Q2-DGX-Spark-00001-of-00002.gguf -ngl 99 -fa on \
    -c 65536 -np 4 --cont-batching --host 0.0.0.0 --port 8080

# single long-context chat
./build/bin/llama-cli -m GLM-5.3-Flash-Q2-DGX-Spark-00001-of-00002.gguf -ngl 99 -fa on -c 65536

• Keep the file on the internal NVMe; the first load reads 79 GiB. Stop other GPU work first.
• Memory on the Spark: 79.1 GiB weights + ~1 GiB per 16 K tokens of context (KDA layers keep a constant state; the DSA layers use a compact cache) + a few GiB of compute buffers. -c 65536 leaves ~30 GiB free; -np 4 -c 65536 (4 × 16 K) is a good multi-user setting.
• Thinking is on by default (the GLM template opens <think>). --reasoning-budget 0 disables it, --reasoning-budget 4096 caps it, --chat-template-kwargs '{"reasoning_effort":"low"}' selects the template's low / high / max effort. The server returns thinking separately as reasoning_content; tool calls come back as OpenAI tool_calls.
• Expected decode speed on a Spark: the same class as the stock Flash Q2 — the per-token work is identical (8 experts + the same 8-bit attention). Decode is bound by the 273 GB/s LPDDR5X (about 11 GB of weights per token), so roughly 15–20 t/s single-stream; batching several sessions gives more aggregate throughput. Not measured by the author on a Spark; see the B200 table below for relative numbers.
• Two Sparks (ConnectX link) are not needed for this file; it is a single-Spark model. For the 744B two-Spark build, see GLM-5.3-GGUF-DGX-Spark.
• Also runs on: 128 GB Apple Silicon (cmake -B build without CUDA), discrete NVIDIA GPUs with ≥ 90 GB, or partially offloaded (-ngl N) on smaller cards.

---

Speed reference

llama.cpp, CUDA, one B200, model resident (llama-bench / llama-batched-bench, 256-token prompts, 128 generated tokens, flash attention):

---

Quality

SLIM stage (4-bit build with the same 256-expert selection, under vLLM on one B200): HumanEval 97.6 · C-Eval 89.4 · GPQA-Diamond 77.3 (low effort) · AIME 2025 74.2 · MMMU 76.1 · BFCL Non-Live 87.7 / Live 80.5 / multi-turn 73–75 — the pruning itself costs little.

Q stage (this 2-bit file) — DwarfStar's ds4-eval harness (GPQA Diamond, SuperGPQA, AIME 2025 interleaved; thinking on, 16 000-token budget, greedy), first 40 core cases, compared with the 2-bit 744B SLIM-Q E192 build on the same cases:

An imatrix-guided build of the same layout scored 31 / 40 with twice the reasoning tokens, so this weight-energy-importance build is the one published. Qualitative checks: correct bilingual common-sense, medical and coding answers; no Chinese–English mixing.

Choosing between the two Spark builds: this Flash build closes more reasoning cases within budget and fits one Spark; the 744B E192 build keeps a clear edge on language modelling of agent/SWE trajectories and code (held-out PPL 6.2 vs 12.4 on agent traces, 3.1 vs 4.6 on SWE traces, 2.96 vs 3.17 on code) and needs two Sparks, a 180 GB GPU, or a big Mac.

---

What is in the file

45 layers (3 dense + 42 MoE) · KDA linear attention with DSA every fourth layer (k-pool indexer) · hyper-connections · top-8 of 256 routed experts + 1 shared · 154 880-token vocabulary · GLM 5.3 chat template with tool calling. Plain llama.cpp GGUF (architecture glm5-next). No MTP block; text model (no vision projector).

Shards

Total 84,929,450,144 bytes (79.1 GiB). Split with llama-gguf-split --split-max-size 45G; to get a single file: llama-gguf-split --merge GLM-5.3-Flash-Q2-DGX-Spark-00001-of-00002.gguf out.gguf.

---

Limitations

• 2-bit routed experts on top of an 11 % expert pruning. Chat, coding and agent use are the target; expect a drop on the hardest maths/science reasoning versus the 4-bit deployment.
• Needs the llama.cpp glm5-next branch until PR #27773 is merged. Two quirks of that branch are patched in llamacpp-pr27773-glm5next-lenient.patch (in the tooling repository): tolerate u64 metadata, and return a reply verbatim instead of HTTP 500 when max_tokens cuts it mid-UTF-8-character. Neither is required to run the model.
• No MTP head, no vision projector in this file.

---

License and credits

• Weights: MIT (base model); derivative of zai-org/GLM-5.3-Flash.
• SLIM-Q pipeline and expert pruning (E256 selection): AutoTrust AI — the same Sparse-Squared compression recipe as the 744B GLM-5.3-SLIM-E192 behind Guru Turbo 2.0.
• 2-bit quantization recipe and quantizer: DwarfStar (antirez/ds4), on llama.cpp / GGML; glm5-next support by the llama.cpp PR #27773 authors.
• Tooling and this build: https://github.com/yuhai-china/ds4-glm-slim