DeepSeek-V4.1-Flash RANMA EXL3 GGUF

[!WARNING] Requires ranma.cpp ranma_20261013 or newer (github.com/saturnsky/ranma.cpp). These files use fork-specific tensor types (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

ranma.cpp is a llama.cpp fork developed with RDNA4 as its main target. It runs MoE models on a three-tier expert cache: VRAM, host RAM and SSD. It supports EXL3 quantization to keep memory traffic as low as possible at a given quality.

Source and credit

This repository repackages the EXL3 quantization made by Mia-AiLab of deepseek-ai/DeepSeek-V4.1-Flash as GGUF. The weights were not re-quantized: the trellis-coded tensors are carried over bit for bit, only wrapped in the GGUF container and split into shards. All credit for the quantization goes to Mia-AiLab and the ExLlamaV3 project. Thank you.

Each folder has a short README. manifest.json lists the size and SHA-256 of every file. Not included: the vision encoder (no image input) and the checkpoint's MTP layers (speculative decoding uses the DSpark draft model).

Running

1. Download the model folder from this repository and the engram tables (202.8 GB, needed, not in this repository) from SaturnHeaven/DeepSeek-V4.1-Flash-RANMA-Engram-GGUF.
2. Give -m only the first shard, DeepSeek-V4.1-Flash-EXL3-3.0bpw-h6-00001-of-00006.gguf; the other shards are found by name.
3. Give --engram-tables the first file of the table set, DeepSeek-V4.1-Flash-engram-MXFP8-00001-of-00007.gguf.

Context, L1/L2 sizes and the draft model for each hardware configuration are in the ranma.cpp recipes. They are examples: choose for your own use and system.

Benchmarks

ranma_20261018 on a Radeon AI PRO R9700 (32 GB) with layer-major prefill (--layer-mode on) and n-gram drafting (--spec-type ngram-varlen): server decode, warm, one slot, the model's recommended sampling, 12 coding and 17 Korean roleplay requests, and prompt processing of 32768 tokens. The host RAM sizes were reproduced on a 224 GB machine by occupying memory. Settings, the other statistics and memory: benchmark record.

The 16 GB card and the DSpark draft model were last measured at ranma_20261013 (record).

License

Same as the original model: MIT, see the original repository.