DeepSeek Engram

IQ4_NL · GSQ–RCO companion · Runtime companion

The exact shared Engram IQ4_NL tensor file and conversion helper used with Mango's native DeepSeek GSQ setup. This is a runtime dependency, not a standalone chat model.

Benchmark results · Files · Download · Runtime · Provenance · Full collection

Join Infinity on Discord

Get model support, request models and quants, discuss benchmarks, and follow new releases and model improvements.

Join the Infinity community

All model tests, per-question scores, experiment history and scoring methods

Qwen effort study — separate full-404 cohort: medium 82.87/100 versus xhigh 84.41/100; paired difference +1.54 points (95% interval [-1.71, +4.82]). 404 collected questions, 403 valid questions graded per effort; one invalid reference annotation excluded symmetrically. The historical 202-question cross-model comparison below is unchanged. Study, timing and blind subjective results.

Preserved and documented by isilicon / Mango. The original model, modification and quantization remain the upstream creators' work. This repository adds attributed preservation and exact-configuration evaluation, with the selected source bytes unchanged.

At a glance

GB is decimal and GiB is binary. Payload size is disk/storage size, not a claim that the model fits in the same amount of VRAM. Runtime, KV cache, working memory and companions add their own requirements.

Mango evaluation

No standalone Mango quality score has been published for these exact files. This is a companion artifact; the consuming model's result belongs to the full target/runtime configuration.

Related configurations measured separately

These results describe the linked configurations, not this artifact. A change of quant, modification lineage, draft, engine or tokenizer can change the result.

A completed score requires a native preflight, matching frozen prompts, 202 suite answers, 12 agent cases, archive verification and the same reference graders. The benchmark methodology describes the protocol; publisher-side results are retained separately below.

Runtime and compatibility

Install it in the layout expected by the compatible Strata conversion/pack loader. Mango's restoration also recreates the required Engram link. The parent model's task score belongs to the complete target/runtime configuration, not to this tensor file.

No independent quality score, context window or chat endpoint is defined for this component.

Companion artifacts

Files and integrity

Full paths, byte sizes and complete SHA-256/Git blob identities are in SOURCE_MANIFEST.json. Available original README/license/notice files are also preserved. MIRROR_STATUS.json records the earlier source-file verification; MODEL_CARD_STATUS.json records this documentation refresh separately.

Download and use

The commands pin the already-verified mirror weight revision. Documentation can advance on main without changing that preserved weight snapshot. Keep original subfolders and all shards.

python -m pip install -U huggingface_hub

hf download isilicon/mango-deepseek41-gsq-engram \
  DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf \
  convert_ds41_upstream.py \
  --revision 23749573fdb80daf88dd108d93843b6b2ec9799f \
  --local-dir ./weights/deepseek41-gsq-engram

Load example and setup scope

This component has no independent chat launch command. Install it through the consuming model's compatible runtime and companion layout.

Provenance and license

ISTA-DASLab supplies the GSQ/RCO release. The preserved Engram file is 110,595,290,816 bytes; a similarly named Engram with a different hash is not an interchangeable verified backup.

Original notices, base-model conditions and the publisher's stated limitations still apply. Downloadable preservation does not add new license rights. Source-described refusal modification and publisher-side benchmark claims have not been independently re-measured by this Mango task cohort.

Original publisher documentation

The unchanged upstream README and pinned source card retain the original usage guidance, technical discussion, attribution, caveats and citations.

Read the complete original publisher card

Source reference. The following is the original publisher's documentation. Its other quants, full-model files, context claims and benchmark tables can exceed the selection in this mirror. Relative asset/file URLs are qualified to the pinned original source; publisher measurements remain separate from Mango measurements.

DeepSeek-V4.1-Flash-1017 · GSQ-RCO GGUFs

Non-uniform GGUF quantizations of a 763B-parameter, 384-expert MoE, produced with GSQ and RCO.

Update, October 27, 2026. The files now use llama.cpp's own DeepSeek-V4.1 layout and load in llama.cpp b11870 or newer, including images.

• Downloaded the earlier files? You do not need to download the model again. Run convert_ds41_upstream.py on your model folder: it rewrites the files locally into exactly the current ones and fetches only 80 KB. See Updating an existing download.

• New download? The shared Engram file is shard 6 of every build, so link it once per build you use (<ID> is IQ1_M, IQ2_XXS or IQ2_XS; see llama.cpp):

  ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-00006-of-00006.gguf
  

---

Overview

This repository provides GGUF quantizations of DeepSeek-V4.1-Flash-1017 at three sizes: IQ1_M (1.75 bits per weight), IQ2_XXS (2.00) and IQ2_XS (2.25). In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model with 763.2B parameters: 384 routed experts per layer, of which 6 are active per token, and a 1M-token context window. The routed experts dominate the parameter count, so they also dominate the size budget, and the search spends most of its freedom there, assigning a quantization type per layer (not per expert, which GGUF cannot express).

With Strata, every build runs on a single GPU with 12 GB of VRAM. See Usage.

Method summary. GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget. Together they yield a non-uniform GGUF at the requested size.

Both methods were developed at the Deep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.

---

Available files

Each build is a backbone GGUF in five shards, named after its quantization identifier: DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-0000i-of-00006.gguf, i = 1 to 5. Shard 1 holds the metadata only, shards 2 to 5 the transformer weights, 551.9B parameters. The bpw is the average bit-width over these, which is what the search controls.

The Engram file, DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf, holds the model's two Engram lookup tables (196.6B parameters) at IQ4_NL, 4.50 bpw. It is identical for all builds, so it is stored once and needs to be downloaded only once, also when trying several builds. llama.cpp loads it as shard 6 of a build, so each build needs a link to it named ...-<ID>-00006-of-00006.gguf (one command, see llama.cpp). The tables are a lookup, not a matmul weight, so they are held at a fixed type in every build and excluded from the search. They are quantized in the evaluated models, not just the shipped ones.

All three builds run on a single 12 GB GPU with Strata.

The Engram column is the same shared file in every row. The vision projector, mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf (0.97 GB), also serves all builds. GB is 10⁹ bytes.

The IQ2_XXS build is the stronger operating point: it matches the base model on all three benchmarks and so reaches 100.0% of its task average (93.54). The IQ1_M build trades 1.2 points of task average for 17.5 GB less.

Memory requirements

Only the backbone holds transformer weights. The Engram tables are read sparsely, one row per token, so they can stay memory-mapped on disk and do not need to occupy VRAM or RAM. With Strata, the backbone does not need to fit in VRAM either: every build runs on a 12 GB GPU, with the rest of the model held in system memory. Keep the Engram file on an SSD so the paging stays cheap.

---

Results

All models are evaluated against the base model deepseek-ai/DeepSeek-V4.1-Flash-1017 as released (FP8 dense weights, FP4 routed experts). We report three reasoning and generation benchmarks: AIME25, GPQA-Diamond, and LiveCodeBench v6. Task average is the mean of the three; recovery is the task average relative to the base model.

All numbers in this card were measured at reasoning effort 50.

IQ2_XXS (2.00 bpw) keeps 100.0% of the base model's task average: it matches the base model on AIME25 (100.00), GPQA-Diamond (90.91) and LiveCodeBench v6 (89.71). IQ2_XS (2.25 bpw) matches it on AIME25 and LiveCodeBench v6 and scores 90.40 on GPQA-Diamond, 99.8% overall. The smallest build, IQ1_M (1.75 bpw), also scores 100.00 on AIME25 and retains 98.7% overall.

---

Usage

Runtime support. llama.cpp supports DeepSeek-V4.1, including images, since release b11870 (October 25, 2026), and the files here use its layout since October 27, 2026. Downloads from before that date can be converted locally, see Updating an existing download. The GGUFs are also supported in Strata.

Strata Engine

Strata runs these GGUFs on a single NVIDIA GPU with 12 GB of VRAM, with a one-click installer and an OpenAI-compatible endpoint on localhost. The builds were optimized together with Strata for local inference.

llama.cpp

Requires llama.cpp b11870 or newer.

# download one build and the shared Engram file (requires: pip install -U "huggingface_hub[cli]")
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "*IQ2_XXS*" "*Engram*" --local-dir .

# the Engram file is shard 6 of every build
ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00006-of-00006.gguf

llama-cli -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
  -p "Explain mixed-precision quantization." -ngl 99

-m takes the first shard; llama.cpp opens shards 2 to 6 from the same directory. To try another build, download only its shards (--include "*IQ1_M*") and add its link. On Windows, mklink /H <link> DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf makes the link; convert_ds41_upstream.py, run on the folder, also creates missing links.

The Engram table stays memory-mapped on disk and is read one row per token (llama.cpp's default --lazy-mode auto does this for tensors over 4 GiB), so it needs neither VRAM nor RAM. See Memory requirements.

Images

Download the vision projector and pass it with --mmproj:

hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "mmproj-*" --local-dir .

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
  --mmproj mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf -ngl 99

With llama-mtmd-cli, also pass --jinja so that the chat template embedded in the GGUF is used.

Chat template and reasoning effort

The base repository ships no Jinja chat template; its prompt format is defined by DeepSeek's encoder, encoding/encoding.py. The GGUFs embed a template adapted from Unsloth's DeepSeek-V4.1 release, checked against that encoder for text chat, tool use, and image formatting. Reasoning effort accepts an integer from 1 to 100 (also a numeric string), or low, high, max (50, 75, 100). The default effort is DeepSeek's high (75), and effort applies only in thinking mode. Structured image blocks, including images in tool results, retain their image placeholders. Image inference needs the vision projector, see Images.

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf -ngl 99 \
  --reasoning on --reasoning-effort 50

Effort 50 is the setting all results in this card were measured with. Per request, set "reasoning_effort": "50" in the chat completion body (a string, also for numbers). --reasoning off disables thinking.

Updating an existing download

Files downloaded before October 27, 2026 (four shards per build and a separate DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf) were written for an earlier llama.cpp branch and do not load in llama.cpp. convert_ds41_upstream.py turns them into the current files of this repository, byte for byte, without downloading the model again. It needs Python 3.8+ and no extra packages, and fetches only two small files of this repository (chat_template.jinja and exp_probs_b_vl.f32, 80 KB).

cd /path/to/downloaded/model-folder
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF convert_ds41_upstream.py --local-dir .
python3 convert_ds41_upstream.py .

It converts every build in the folder and the mmproj, and writes the new backbone shards next to the old ones; every written tensor is read back and compared with its source. The Engram file is converted in place: only its header changes, its 110 GB of tables stay as they are, and the script adds the shard-6 links. The free space needed is the size of the new shards (121 to 155 GB per build). With --delete-old, the old shards of a build are deleted as soon as its new ones are verified, so space for one build suffices. The new first shards take the chat template from chat_template.jinja, so patch_chat_template.py is not needed.

---

Quantization procedure

1. Per-tensor database. Each weight tensor is quantized at every candidate GGUF quantization type with GSQ, yielding a searchable database of quantized tensor variants.
2. RCO search. The budget-constrained Riemannian search assigns one quantization type per tensor such that the backbone's average bit-width meets the target.
3. Assembly. The selected per-tensor variants are stitched into a standard GGUF file.

The dense tensors (attention, shared experts, embeddings, output head) and the routed-expert matrices draw on separate quantization ladders, because the routed experts sit at a much lower bit-width than the dense path at any given budget: the dense tensors choose among Q5_K, Q6_K and Q8_0, the routed experts among types from IQ1_S to IQ3_S. The search's teacher is the released checkpoint itself, with the routed experts in their native FP4, and its calibration data was generated by the model.

Reference implementations: GSQ at IST-DASLab/GSQ and RCO at IST-DASLab/RCO.

Reproducibility artifacts

Each released build ships the files needed to audit how it was built:

---

Citation

If you use these models or methods, please cite both papers:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

---

Acknowledgements

We thank Verda and Scientific Computing at the Institute of Science and Technology Austria for providing the compute resources used to produce these models.

---

License

These quantized weights inherit the license of the base model (DeepSeek-V4.1-Flash-1017, MIT). The GSQ-RCO tooling is released by the Deep Algorithms and Systems Lab under its repository license.

_{Built with GSQ and RCO at the Deep Algorithms and Systems Lab · Institute of Science and Technology Austria}

Preservation record

Selected source bytes are verified against pinned metadata; the original eight graded mirrors also passed the previously recorded anonymous 1 MiB weight-prefix checks. This is not a full independent-device restore/load test. Hugging Face repository copies share a provider and do not guarantee permanent storage.

Collection · Source manifest · Original documentation