GLM-5.3-Flash-EXL3-Spark RANMA EXL3 GGUF

[!WARNING] Requires ranma.cpp ranma_20261007 or newer (github.com/saturnsky/ranma.cpp). These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

[!NOTE] No benchmark numbers are published for this model.

What this is

This repository repackages the EXL3 weights of 0xSero/GLM-5.3-Flash-EXL3-Spark into GGUF files that ranma.cpp can read. The source artifact takes the 2.05 bpw quantization made by turboderp and raises twelve of the forty-five MoE layers (layers 3, 32, 33 and 36–44, all 288 experts of each) to 3.05 bpw, for an average of about 2.31 bpw; every other layer, the attention and embedding weights, the tokenizer and the MTP layer are byte-for-byte the base. The weights in this repository are not re-quantized either: the EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container (with the fork's EXL3 tensor types) and split into shards. All credit for the quantization goes to turboderp for the base and to 0xSero for the twelve upgraded layers, and to the ExLlamaV3 project. Thank you.

Produced with ExLlamaV3 quantization version 1.4.4 (codebook mul1, calibration 250 rows x 2048). h5 is the head bits (output layer).

The multi-token-prediction (MTP) layer of the checkpoint is included in the same file; see Running.

Not included

In llama.cpp the vision encoder is a separate mmproj file, so it can be added later without changing the files here.

Files

The quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors (llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma; the rest are found automatically. manifest.json lists the size and SHA-256 of every file.

File names: GLM-5.3-Flash-EXL3-Spark-0000K-of-00003.gguf.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the example below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only; there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement). The first shard is enough for -m; the other shards are found by name.

The command and values below are an example only. Pick them for your own use and system: the context length, the split between L1 and L2, whether to use MTP, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and workload works best).

export GGML_CUDA_HOST_DIRECT=1
export GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-Spark/GLM-5.3-Flash-EXL3-Spark-00001-of-00003.gguf \
  -c 65536 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none \
  --expert-cache-mode exclusive --expert-l1-mib 20480 --expert-l2-mib 92160 \
  --expert-profile-dir ./expert-profiles

The routed experts of this build take 83.1 GiB in total, so an L1 + L2 at or above that reads nothing from the SSD.

Details of every option: expert-cache.md, expert-cache-exclusive.md, expert-cache-l2.md, expert-cache-joint.md, host-direct-moe.md, spec-smart.md, gpu-heartbeat.md, exl3.md.

License

Same as the original model: MIT, see the original repository.