DeepSeek-V4.1-Flash-1017 · GSQ-RCO GGUFs

Non-uniform GGUF quantizations of a 763B-parameter, 384-expert MoE, produced with GSQ and RCO.

---

Overview

This repository provides GGUF quantizations of DeepSeek-V4.1-Flash-1017 at three sizes: IQ1_M (1.75 bits per weight), IQ2_XXS (2.00) and IQ2_XS (2.25). In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model with 763.2B parameters: 384 routed experts per layer, of which 6 are active per token, and a 1M-token context window. The routed experts dominate the parameter count, so they also dominate the size budget, and the search spends most of its freedom there, assigning a quantization type per layer (not per expert, which GGUF cannot express).

With Strata, every build runs on a single GPU with 12 GB of VRAM. See Usage.

Method summary. GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget. Together they yield a non-uniform GGUF at the requested size.

Both methods were developed at the Deep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.

---

Available files

Each build is a backbone GGUF in four shards, named after its quantization identifier: DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-0000i-of-00004.gguf. The backbone holds the transformer weights, 551.9B parameters, and its bpw is the average bit-width over these, which is what the search controls.

The Engram file, DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf, holds the model's two Engram lookup tables (196.6B parameters) at IQ4_NL, 4.50 bpw. It is identical for all builds, so it is stored once and needs to be downloaded only once, also when trying several builds. The tables are a lookup, not a matmul weight, so they are held at a fixed type in every build and excluded from the search. They are quantized in the evaluated models, not just the shipped ones. Keep the Engram file in the same directory as the backbone shards: it is loaded from there.

All three builds run on a single 12 GB GPU with Strata.

The Engram column is the same shared file in every row. GB is 10⁹ bytes.

The IQ2_XXS build is the stronger operating point: it matches the base model on all three benchmarks and so reaches 100.0% of its task average (93.54). The IQ1_M build trades 1.2 points of task average for 17.5 GB less.

Memory requirements

Only the backbone holds transformer weights. The Engram tables are read sparsely, one row per token, so they can stay memory-mapped on disk and do not need to occupy VRAM or RAM. With Strata, the backbone does not need to fit in VRAM either: every build runs on a 12 GB GPU, with the rest of the model held in system memory. Keep the Engram file on an SSD so the paging stays cheap.

---

Results

All models are evaluated against the base model deepseek-ai/DeepSeek-V4.1-Flash-1017 as released (FP8 dense weights, FP4 routed experts). We report three reasoning and generation benchmarks: AIME25, GPQA-Diamond, and LiveCodeBench v6. Task average is the mean of the three; recovery is the task average relative to the base model.

All numbers in this card were measured at reasoning effort 50.

IQ2_XXS (2.00 bpw) keeps 100.0% of the base model's task average: it matches the base model on AIME25 (100.00), GPQA-Diamond (90.91) and LiveCodeBench v6 (89.71). The smaller IQ1_M build (1.75 bpw) also scores 100.00 on AIME25 and retains 98.7% overall.

---

Usage

Runtime support. As of October 23, 2026, llama.cpp does not support DeepSeek-V4.1, so these GGUFs do not run in llama.cpp yet. They are supported in Strata.

Strata Engine

Strata runs these GGUFs on a single NVIDIA GPU with 12 GB of VRAM, with a one-click installer and an OpenAI-compatible endpoint on localhost. The builds were optimized together with Strata for local inference.

llama.cpp

The commands below apply once llama.cpp supports DeepSeek-V4.1 (see Runtime support).

# download one build and the shared Engram file (requires: pip install -U "huggingface_hub[cli]")
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "*IQ2_XXS*" "*Engram*" --local-dir .

llama-cli -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00004.gguf \
  -lm mmap --lazy-mode on \
  -p "Explain mixed-precision quantization." -ngl 99

-m takes the first shard; llama.cpp opens the other three shards and the Engram file from the same directory. To try another build, download only its shards (--include "*IQ1_M*"). -lm mmap --lazy-mode on keeps the Engram tables memory-mapped on disk instead of loading them into memory. See Memory requirements.

Chat template and reasoning effort

The base repository ships no Jinja chat template; its prompt format is defined by DeepSeek's encoder, encoding/encoding.py. The GGUFs embed a chat template that reproduces this encoder exactly, including the reasoning effort: an integer from 1 to 100, or low, high, max (50, 75, 100). The default is high, and the effort applies only in thinking mode.

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00004.gguf -ngl 99 \
  --reasoning on --reasoning-effort 50

Effort 50 is the setting all results in this card were measured with. Per request, set "reasoning_effort": "50" in the chat completion body (a string, also for numbers). --reasoning off disables thinking.

---

Quantization procedure

1. Per-tensor database. Each weight tensor is quantized at every candidate GGUF quantization type with GSQ, yielding a searchable database of quantized tensor variants.
2. RCO search. The budget-constrained Riemannian search assigns one quantization type per tensor such that the backbone's average bit-width meets the target.
3. Assembly. The selected per-tensor variants are stitched into a standard GGUF file.

The dense tensors (attention, shared experts, embeddings, output head) and the routed-expert matrices draw on separate quantization ladders, because the routed experts sit at a much lower bit-width than the dense path at any given budget: the dense tensors choose among Q5_K, Q6_K and Q8_0, the routed experts among types from IQ1_S to IQ3_S. The search's teacher is the released checkpoint itself, with the routed experts in their native FP4, and its calibration data was generated by the model.

Reference implementations: GSQ at IST-DASLab/GSQ and RCO at IST-DASLab/RCO.

Reproducibility artifacts

Each released build ships the files needed to audit how it was built:

---

Citation

If you use these models or methods, please cite both papers:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

---

Acknowledgements

We thank Verda and Scientific Computing at the Institute of Science and Technology Austria for providing the compute resources used to produce these models.

---

License

These quantized weights inherit the license of the base model (DeepSeek-V4.1-Flash-1017, MIT). The GSQ-RCO tooling is released by the Deep Algorithms and Systems Lab under its repository license.

_{Built with GSQ and RCO at the Deep Algorithms and Systems Lab · Institute of Science and Technology Austria}