DeepSeek-V4-Flash-0731 RANMA EXL3 GGUF

[!WARNING] Requires ranma.cpp ranma_20261005 or newer (github.com/saturnsky/ranma.cpp). These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

[!NOTE] Status: loading and generation work, but the DeepSeek-V4-Flash path in ranma.cpp has not been performance-optimized yet. Expect it to be slower than the Qwen3.8-Flash-Next EXL3 files. No benchmark numbers are published for this model.

What this is

This repository repackages the EXL3 quantization made by turboderp of deepseek-ai/DeepSeek-V4-Flash-0731 into a GGUF that ranma.cpp can read.

The weights were not re-quantized. The EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container (with the fork's EXL3 tensor types) and split into shards. All credit for the quantization goes to turboderp and the ExLlamaV3 project. Thank you.

Source (EXL3, by turboderp): turboderp/DeepSeek-V4-Flash-0731-exl3, branch 3.04bpw (ExLlamaV3 quantization version 1.3.0, codebook mul1, head bits 6, calibration 250 rows x 2048).

Not included

This file holds the main model only. The source checkpoint's mtp.safetensors (7.1 GiB: the DSpark draft model, three layers plus its prediction heads) is not carried over: the converter does not handle it yet. Speculative decoding with it is not available from this repository. In llama.cpp a draft model is a separate GGUF, so it can be added later without changing the files here.

Files

Each quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors (llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma. manifest.json lists the size and SHA-256 of every file.

File names: DeepSeek-V4-Flash-0731-EXL3-3.04bpw-h6-0000K-of-0000N.gguf.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the examples below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only; there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement). The first shard is enough for -m; the other shards are found by name.

Sizing L1 and L2

• L1. L1 = VRAM - (memory the cache does not cover) - margin. What the cache does not cover is the dense weights, the KV cache and the compute buffers; the KV cache grows with the context (little for this model, see the example command). Reference: for Qwen3.8-Flash-Next about 8.3 to 9.5 GiB of device memory were outside the cache at context 65536; for this DeepSeek model the verification run with L1 = 16 GiB peaked at 23.7 GiB dedicated VRAM, so measure your own context size. Keep a margin of a few GiB, more if the GPU also drives a display. The benchmark budgets were deliberately conservative; a larger L1 puts more experts in VRAM. The L1 budget includes the cache's own tables and overhead, so it is not exactly the expert bytes.
• L2. With -1, the RAM used by the experts is about the model size minus L1 (exclusive mode); the KV cache, prompt cache and buffers come on top. With a finite L2 the cache's host part is bounded by it. Set it to the RAM you can spare minus what your other programs use. The server refuses to load if the host requirement is above the physical memory installed, but it does not look at how much is free at that moment.
• If L1 + L2 covers all experts, nothing is read from the file; each expert that has to come from the SSD costs speed.

The commands and values below are examples only. Pick them for your own use and system: the context length (a long roleplay needs a large -c, a short chat or a coding session can use much less and leave VRAM to L1), the split between L1 and L2, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and workload works best).

Example command (256K context)

The KV cache grows with the context, but this model keeps only a short sliding window of raw cells plus compressed caches (1 cell per 4 or 128 tokens), so it is small: about 1.8 GiB at -c 262144 (f16, estimated from the cache layout). The L1 and L2 values are the ones this conversion was verified with (L1 16 GiB exclusive, finite L2 40 GiB, the remaining experts read from the SSD, no MTP, peak private memory about 65 GiB, roughly L2 plus 24 GiB; dedicated VRAM peak 23.7 GiB at the verification context). Checked at -c 262144 -np 1 on an R9700 (32 GB): load and a short generation worked; dedicated VRAM peak 23.0 GiB, private memory peak 64.0 GiB. Adjust L1 and L2 to your machine as described above.

32 GB VRAM + 128 GB RAM:

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.04bpw-h6\DeepSeek-V4-Flash-0731-EXL3-3.04bpw-h6-00001-of-00004.gguf ^
  -c 262144 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 16384 --expert-l2-mib 40960 ^
  --expert-profile-dir .\expert-profiles

Details of every option: expert-cache.md, expert-cache-exclusive.md, expert-cache-l2.md, host-direct-moe.md, spec-smart.md, gpu-heartbeat.md.

License

Same as the original model: MIT, see the original repository.