GLM-5.3-Flash GGUF for the Framework Desktop (Gorgon Halo)

A 3.84 bpw GGUF of zai-org/GLM-5.3-Flash (320.8B total parameters, MoE with 288 experts, top-8), sized to run fully on the GPU of a 192 GB AMD Ryzen AI Max 400 ("Gorgon Halo") Framework Desktop.

• Format: llama.cpp glm5-next (llama.cpp PR #27773). It loads in current llama.cpp master and in ROCmFPX.
• Size: 143.5 GiB (154.1 GB) in 4 shards. Pass the first shard to -m; llama.cpp finds the rest.
• The MTP (next-token prediction) layer is included, but llama.cpp does not run GLM-5.3 MTP drafting yet, so plain decoding is used.

Files

Quantization recipe

Routed experts carry almost all of the weights, so they get the low-bit types; everything that every token passes through stays at 5 bits or more.

Built from Unsloth's GLM-5.3-Flash UD-IQ4_XS GGUF and its importance matrix (glm-imatrix.gguf), requantizing the attention and dense tensors to Q5_K and the routed experts as above. The file was then converted from the pre-merge glm5next layout to the merged glm5-next layout: keys renamed, indexer.kpool_select_tail = true and indexer.types (all full) taken from the official text_config, and the gate/up experts stored as separate tensors. The conversion is a byte copy of the weights.

Quality

• KL divergence against Unsloth's UD-IQ4_XS GGUF (the file this mix was made from): mean KLD 0.040, same top-1 token 93.1% (llama-perplexity --kl-divergence, 16 x 1024-token chunks, Vulkan).
• Vulkan vs CPU on this exact file (12K tokens, ctx 2048): mean KLD 0.018, same top-1 96.3%. This measures backend numerics, not quantization loss.

Speed on the Framework Desktop (Gorgon Halo)

AMD Ryzen AI Max+ PRO 495 / Radeon 8065S, 192 GB LPDDR5X, Ubuntu 26.04, kernel 7.0, Mesa 26.0.8 RADV, Vulkan backend. ROCmFPX main (GLM-5.3 support from #34, prefill work from #35 and #36). llama-bench -fa 1 -b 2048 -ub 2048, 3 runs:

These are local measurements on one machine, not leaderboard results. Plain llama.cpp master runs the file at the same decode speed; prefill is lower there because it does not have #35 and #36.

How to run

1. Let the GPU use the memory

The weights need about 144 GiB of GPU-visible memory plus the KV cache. On a 192 GB machine, raise the GTT limit with kernel parameters (this example gives the GPU 176 GiB) and reboot:

ttm.pages_limit=46137344 ttm.page_pool_size=46137344

On Ubuntu, add them to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, run sudo update-grub, and reboot. Check with:

cat /sys/class/drm/card*/device/mem_info_gtt_total   # about 189000000000 bytes

Keep the BIOS GPU memory carve-out at its minimum: the GPU uses GTT, and memory carved out as VRAM is no longer available to it as GTT.

2. Build ROCmFPX (or llama.cpp) with Vulkan

sudo apt install build-essential cmake libvulkan-dev glslc mesa-vulkan-drivers
git clone https://github.com/ROCmFPX/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-cli

Stock llama.cpp (https://github.com/ggml-org/llama.cpp) builds the same way and also runs this file, with slower prefill.

3. Download

hf download cafonez/GLM-5.3-Flash-Gorgon-GGUF --local-dir GLM-5.3-Flash-Gorgon-GGUF

4. Serve

export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 GGML_VK_MAX_NODES_PER_SUBMIT=2000 GGML_VK_ALLOW_GRAPHICS_QUEUE=1
./build/bin/llama-server \
  -m GLM-5.3-Flash-Gorgon-GGUF/GLM-5.3-Flash-Q5K-IQ3S-00001-of-00004.gguf \
  -dev Vulkan0 -ngl 999 -fa on \
  -c 131072 -b 2048 -ub 2048 -np 1 \
  --jinja --temp 0.7 --top-p 0.95 \
  --host 127.0.0.1 --port 8080

• --jinja is required: the chat template handles thinking and tool calls. Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} in a request to turn it off, or cap it with --reasoning-budget N.
• -c: 131072 fits comfortably with 176 GiB of GTT. The model supports up to 1M tokens; larger contexts need more memory, and -ub 1024 lowers the compute buffers if you run short.
• -ub 2048 gives the best prefill; decode is the same at any -ub.
• The three GGML_VK_* variables were set for all measurements above.

Then use any OpenAI-compatible client against http://127.0.0.1:8080/v1.

Credits

• Model: Z.ai GLM-5.3-Flash (MIT).
• Importance matrix and the source GGUF: Unsloth.
• llama.cpp GLM-5.3 support: PR #27773 and its contributors.