Abyss (engine) | Base model | Unpruned LDLQ 132G / 139G | trench-gguf | Report a result | 日本語は下
132G: a mixed-type file - the experts are IQ2_XXS 45%, IQ1_S 30%, IQ2_XS 14%, IQ2_S 11% (by bytes, as in 132G), 1.95 bits a weight on average, about IQ2_XXS.A general-purpose pruned build of deepseek-ai/DeepSeek-V4.1-Flash-1017 (MIT) - not code-only. 256 of 384 experts per layer are kept, chosen by measured routing on chat, math, tool-call, code and multilingual text. 207 GB instead of the 276 GB of our unpruned LDLQ 132G, and faster on one 32 GB GPU plus a 96 GB RAM PC. Runs in Abyss and llama.cpp.
BETA: measured on one PC only (RTX 5090 32 GB, RAM tier capped at 60 GB with STRATA_GLM_RAM_GB=60). Reports from other GPUs / RAM sizes are welcome (see Feedback).
JA256/DeepSeek-V4.1-Flash-1017-JA256-0000N-of-00006.gguf, 6 shards. Point your engine at -00001-of-00006.gguf.
SHA256SUMS is in the folder.
| shards 1-5 (dense + experts) | shard 6 (Engram table) | total | |
|---|---|---|---|
| JA256 | 96.5 GB (experts 88.4 GB) | 110.6 GB (IQ4_NL) | 207.1 GB |
expert_count = 256; nothing re-quantized). Which experts are kept differs per layer.| repo | for a PC with | download | note |
|---|---|---|---|
| DeepSeek-V4.1-Flash-1017-IQ2_XXS-32G96G-GGUF | 32 GB GPU + 96 GB RAM | 275.8 / 283.0 GB | all 384 experts (folders 132G / 139G) |
| this repo | 32 GB GPU + 96 GB RAM; also 12 GB GPU + 128 GB RAM | 207.1 GB | 256 of 384 experts kept (folder JA256) |
Earlier repo name: DeepSeek-V4.1-Flash-1017-Compact-32G96G-GGUF (old links redirect here).
Names: the quant in the repo name is the nearest llama.cpp type to the experts' average bits; most files mix several types (listed above), so they are not a stock quant of that name. E<n> = experts kept per layer (only when some were removed).
| experts per layer | download | KL (8 seqs) | same top token | decode, 3 coding tasks | Japanese | |
|---|---|---|---|---|---|---|
JA256/ (here) |
256 / 384 | 207.1 GB | 0.383 | 75.2% | 87.0 tok/s | OK |
LDLQ 132G |
384 / 384 | 275.8 GB | 0.194 | 81.1% | 67.5 tok/s | OK |
LDLQ 139G |
384 / 384 | 283.0 GB | 0.182 | 81.6% | 57.9 tok/s | OK |
| ISTA GSQ-RCO Coder (reference) | 192 / 384 | 182.7 GB | 0.582 | 69.3% | 97.0 tok/s | Chinese |
| ISTA GSQ-RCO IQ2_XXS (reference) | 384 / 384 | ~249 GB | 0.314 | 76.8% | - | - |
How to choose:
Abyss v1.3.101 or later. The one-command setup does not install this repo yet, so set up the engine with the V4.1 setup once, then add this file by hand:
git clone https://github.com/zephel01/Abyss.git ~/llm/apps/Abyss
cd ~/llm/apps/Abyss
./setup-deepseek.sh --model v41 # builds the engine; decline the 132G download and it stops there, which is enough
.venv/bin/pip install -U huggingface_hub
.venv/bin/hf download zephel01/DeepSeek-V4.1-Flash-1017-E256-IQ2_XXS-32G96G-GGUF --include "JA256/*" --local-dir ../Abyss-data/models/v41-ja256
.venv/bin/python tools/iq_pack.py --compat-bf16 \
--gguf ../Abyss-data/models/v41-ja256/JA256/DeepSeek-V4.1-Flash-1017-JA256-00001-of-00006.gguf \
--out ../Abyss-data/models/v41-ja256/JA256/pack
cat > abyss-dsv41-ja256.json <<EOF
{"exe": "$PWD/build/strata",
"args": ["--glm-pack", "$PWD/../Abyss-data/models/v41-ja256/JA256/pack", "--max-context", "65536", "--prefill", "auto"],
"cwd": "$PWD", "tokenizer": "$PWD/../Abyss-data/models/v41-ja256/JA256/pack/tokenizer",
"model_name": "deepseek-v4.1-flash-1017", "gpu": [0], "sampling": {"temperature": 1.0, "top_p": 0.95},
"port": 8082, "log": "$PWD/abyss-dsv41-ja256.log",
"env": {"STRATA_GLM_RAM_GB": "60", "STRATA_GLM_DISK_READERS": "8", "STRATA_GLM_TAIL_SKIP": "4",
"STRATA_GLM_TAIL_SKIP_DISK": "1", "STRATA_GLM_ROUTE_AFFINITY_RAM": "0.08"}}
EOF
systemd-run --user --scope -p MemoryMax=88G .venv/bin/python serve/server.py --engine strata \
--config abyss-dsv41-ja256.json --port 8082
env block holds the everyday settings the speeds below were measured with. Drop the two TAIL_SKIP entries
for exact routing. On a 12 GB GPU + 128 GB RAM PC, see that section below.llama.cpp (arch deepseek41, upstream ac8e1694d of 2026-10-29) runs JA256 as is (the expert count is read
from the file). Checked on the RTX 5090 (32 GB): correct answers to a Japanese question and a Python task, 15-20 tok/s
(132G: 10-11 tok/s at --n-cpu-moe 38).
hf download zephel01/DeepSeek-V4.1-Flash-1017-E256-IQ2_XXS-32G96G-GGUF --include "JA256/*" --local-dir V41-JA256
llama-server -m V41-JA256/JA256/DeepSeek-V4.1-Flash-1017-JA256-00001-of-00006.gguf -ngl 99 --n-cpu-moe 36 -c 4096 --jinja
KL against the unquantized model (top-256 log-probs of DeepSeek-V4.1-Flash-1017, 8 held-out sequences x 1,023 tokens), Abyss engine, exact routing: see the table above (per-sequence top-1 agreement of JA256: 72-79%).
KL is a guide only; real work (Abyss server, everyday settings, temperature 1.0 / top_p 0.95):
| JA256 | LDLQ 132G | ISTA Coder | |
|---|---|---|---|
| 10 hard coding tasks (hidden tests), solved | 8/10 (t104 solved in 2 of 2 reruns; t046 failed 3 of 3) | 10/10 | 9/10 (t046 failed) |
| quality score of the solved tasks (graded) | 80.5 | 77.6 | 77.5 |
| 8K / 32K prompt, find the needle | OK / OK | OK / OK | OK / OK |
| 7-turn tool-call coding session, tests pass | yes, final report in Japanese | yes | yes, final report in Chinese |
| Japanese (3 prompts), no mixed scripts | OK | OK | Chinese (no kana, one loop) |
| loops / illegal memory accesses | 0 / 0 | 0 / 0 | 1 / 0 |
The one task JA256 keeps failing (t046, object lifetimes in a dependency-injection container) is also the one the Coder fails: a sign of what pruning costs on hard code. If coding is your main use, take 132G / 139G.
Abyss v1.3.101, one PC: RTX 5090 (32 GB) on OCuLink PCIe Gen4 x4, Ryzen AI Max+ 395 (LPDDR5X, Radeon 8060S iGPU computes
part of the RAM-side experts), RAM tier capped at 60 GB, NVMe SSD. Everyday settings: DSpark drafter (Lucebox
Q2K-Q4K), STRATA_GLM_TAIL_SKIP=4, STRATA_GLM_TAIL_SKIP_DISK=1, STRATA_GLM_ROUTE_AFFINITY_RAM=0.08.
| decode tok/s | JA256 | LDLQ 132G | ISTA Coder |
|---|---|---|---|
| 3 coding tasks | 87.0 | 67.5 | 97.0 |
| 10 hard tasks (mean) | 75.7 | 58.0 | 87.1 |
| 32K prompt: prompt processing | 1,833 | 865 | 1,509 |
| 32K prompt: decode after it | 63.1 | 43.4 | 76.8 |
JA256's experts (88 GB) nearly fit in VRAM + the 60 GB RAM tier: 0.1 SSD reads a token against ~1 for 132G. Other GPUs and 64 GB RAM are not measured with this file; reports welcome.
Abyss v1.3.120 (+ the IQ1_S prompt fix of the next release for ISTA), the same PC set up as a 12 GB card
(STRATA_GLM_VRAM_GB=12) with no iGPU (STRATA_GLM_IGPU=0), the RAM tier at 90 GB (80 for ISTA), exact settings only
(no tail skip, no route affinity), ./abyss.sh --bench:
| experts | KL (8 seqs) | decode | 8K prompt | |
|---|---|---|---|---|
| JA256, no drafter | 256 | 0.383 | 27.4 tok/s | 375 tok/s |
| JA256, DSpark drafter | 256 | 0.383 | 29.5 tok/s | 26 tok/s |
| ISTA GSQ-RCO IQ1_M, no drafter | 384 | 0.384 | 26.7 tok/s | 234 tok/s |
../Abyss-tools/abyss.sh --bench --deep in Abyss, post abyss-bench.txt with
this issue form
(change the title to your GPU), with your RAM size and SSD.With trench-gguf (tools/prune_experts.py, tools/requant_table.py) and
Abyss's routing counter (STRATA_GLM_USAGE):
prune_experts.py cuts the experts, router rows and biases of LDLQ 132G.requant_table.py re-quantizes the FP8 table of unsloth UD-Q4_K_XL to IQ4_NL on the GPU (11 min).MIT, as the original model. All credit for the model goes to DeepSeek; the non-expert tensors come unchanged from unsloth/DeepSeek-V4.1-Flash-1017-GGUF.
132G から切り出したものです。形式を混ぜたファイルで、エキスパートは IQ2_XXS 45%, IQ1_S 30%, IQ2_XS 14%, IQ2_S 11%(容量の割合、132G の値)、平均 1.95 ビットで、IQ2_XXS くらいです。deepseek-ai/DeepSeek-V4.1-Flash-1017(MIT)の GGUF を、 自作の LDLQ 132G / 139G より小さくして置いています。 GPU 32 GB + RAM 96 GB の PC 向けで、276 GB のダウンロードは重いという方のための版です。
BETA 版です。 測ったのは 1 台だけです(RTX 5090、RAM の置き場を 60 GB に制限)。ほかの GPU や RAM 容量での結果を教えてもらえると助かります。
| 1 層のエキスパート | ダウンロード | KL(8 本) | 1 位の一致 | 生成速度(コーディング 3 問) | 日本語 | |
|---|---|---|---|---|---|---|
JA256/(ここ) |
256 / 384 | 207.1 GB | 0.383 | 75.2% | 87.0 tok/s | OK |
LDLQ 132G |
384 / 384 | 275.8 GB | 0.194 | 81.1% | 67.5 tok/s | OK |
LDLQ 139G |
384 / 384 | 283.0 GB | 0.182 | 81.6% | 57.9 tok/s | OK |
| ISTA GSQ-RCO Coder(参考) | 192 / 384 | 182.7 GB | 0.582 | 69.3% | 97.0 tok/s | 中国語 |
| ISTA GSQ-RCO IQ2_XXS(参考) | 384 / 384 | 約 249 GB | 0.314 | 76.8% | - | - |
./setup-deepseek.sh --model v41 でエンジンを用意したあと、上の英語の節のコマンドでこのファイルを足します(1 行のセットアップにはまだ入っていません)。--n-cpu-moe 36 のとき、日本語の質問と Python の課題に正しく答え、生成は 15〜20 tok/s でした。速さの報告は この issue フォーム に、 品質の報告(132G より弱いと感じたところ、特にコーディングと 6 つの領域以外の言語)は Community タブか Abyss の issue にお願いします。
| リポジトリ | 向けの PC | ダウンロード | メモ |
|---|---|---|---|
| DeepSeek-V4.1-Flash-1017-IQ2_XXS-32G96G-GGUF | GPU 32 GB + RAM 96 GB | 275.8 / 283.0 GB | 384 個すべて(132G / 139G) |
| この版 | GPU 32 GB + RAM 96 GB、GPU 12 GB + RAM 128 GB でも | 207.1 GB | 384 個中 256 個(JA256) |
以前のリポジトリ名は DeepSeek-V4.1-Flash-1017-Compact-32G96G-GGUF です(古い URL はここへ転送されます)。
名前の量子化の型は、エキスパートの平均ビット数にいちばん近い llama.cpp の型です。多くのファイルは複数の型を混ぜているので(上に内訳)、その名前の普通の量子化と同じものではありません。E<数> は層ごとに残したエキスパートの数です(減らしたときだけ付けます)。
JA256/DeepSeek-V4.1-Flash-1017-JA256-00006-of-00006.gguf 111 GBJA256/DeepSeek-V4.1-Flash-1017-JA256-00002-of-00006.gguf 31 GBJA256/DeepSeek-V4.1-Flash-1017-JA256-00003-of-00006.gguf 29 GBJA256/DeepSeek-V4.1-Flash-1017-JA256-00004-of-00006.gguf 27 GBJA256/DeepSeek-V4.1-Flash-1017-JA256-00005-of-00006.gguf 9.7 GBJA256/DeepSeek-V4.1-Flash-1017-JA256-00001-of-00006.gguf 6 MBdeepseek-ai/DeepSeek-V4.1-Flash-1017