DeepSeek-V4.1-Flash-1017 abliterated (rumplestilzken) — lecore vq11 calibrated GGUF (106.8 GB)

A 106.8 GB single-file GGUF of rumplestilzken/DeepSeek-V4.1-Flash-1017-abliterated, the rumplestilzken abliteration of deepseek-ai/DeepSeek-V4.1-Flash-1017, quantized with lecore activation-calibrated 11-bit vector quantization of the routed experts.

Credit: all abliteration work belongs to rumplestilzken (source model). The base model is DeepSeek's DeepSeek-V4.1-Flash-1017. This repo only re-quantizes it.

What is in this build

• Routed experts (557B params): VQ, 11-bit codebooks (2048 entries) everywhere, activation-weighted (calibrated) k-means on 65,536 calibration tokens; 1.39 bpw, weighted cosine 0.909.
• Shared/common experts: lecore VQ11 (11-bit), cosine 0.910.
• Dense skeleton: q4_K (AMD-supported GEMV path) — attention q_a / q_b / kv / output_a / output_b, shared-expert FFN gate / up / down (shexp), indexer.attn_q_b on gated layers, MTP main_proj, and the output head. (Previously fp8; re-emitted as q4_K.)
• Token embedding (token_embd): q4_K.
• engram_wkv: fp8.
• Engram n-gram tables: DeepSeek's original shards, used unchanged (download separately, ~203 GB — see Sidecars and Engram tables).
• File type-scan passed; decode-parity check passed against the source weights.

Files in this repo

One 103.43 GB (96.3 GiB) file of weights.

Sidecars and Engram tables

The 103 GB file is the model, not the whole runtime. Three companion files plus DeepSeek's Engram shards are needed to serve it:

Drop the vision file next to the big GGUF (the launcher points --vision-gguf at that directory) and pass --engram-dir <dir> with the two shards in it. The drafter and skeleton files are the pqhaz build's; other abliterations can use the vision tower and Engram tables but should expect a lower DSpark accept rate with a drafter trained on different weights.

Evaluation

Headline (2k context): PPL 2.82 / top-1 80.5% vs source. Full table (earlier fp8-skeleton emit) below.

Teacher = the unquantized source model (rumplestilzken/DeepSeek-V4.1-Flash-1017-abliterated), same official forward pass. WikiText-2 test set. Tail metrics over the 1,233 reserved token ids in the tokenizer (place_holder / control ids).

For reference, the earlier uncalibrated lecore-vq10sh12 build of the original model: PPL 3.615, KL 0.797 mean / 0.195 median, top-1 74.5% (2k).

Recommended sampling

temperature 0.6, top_p 0.95, min_p 0.05, thinking/reasoning off for agent use. Greedy (temperature 0) also works.

Serving (Windows / Radeon 8060S, native HIP)

Measured on a Radeon 8060S (gfx1151, HIP SDK 7.2, 128 GB unified memory) with the patched ds4-server from staccs/ds4-winhip. Paths are examples.

Start

cd ds4-winhip
.\hip\win\run_ds41.ps1 -Engram D:\models\ds41\deepseek-engram      # OpenAI-compatible API on http://127.0.0.1:8008/v1

run_ds41.ps1 sets these unless already present in the environment:

Load is ~6 minutes to the first answer: the weight copy, then the first request pays ~110 s of one-off kernel and graph warm-up. Do not benchmark the first request. One request is served at a time; a second one queues.

Speed on this hardware (measured on the vq10sh12 build): prefill 135-190 tok/s (real agent prompts vs synthetic), decode 14-17 tok/s. Both are kernel-bound, not format-bound.

Sampling

The earlier uncalibrated vq10sh12 build produced word salad with reserved tokens (e.g. place_holder_no_362) under T=1.0 with no truncation at 14k+ prompts. This build's tail metrics (junk mass, escape mass, top-p stability) are in the Evaluation section. Recommended regardless:

• temperature 0 (greedy), or 0.6-0.7 with top_p 0.95 and min_p 0.05;
• reasoning off for agent use (reasoning_effort: "none" or thinking: {"type":"disabled"}).

Hermes Agent

~/.hermes/config.yaml (Windows: %LOCALAPPDATA%\hermes\config.yaml):

provider: custom            # not "deepseek": that name selects the cloud provider and demands its key
base_url: http://127.0.0.1:8008/v1
api_key: local
model: local-ds41           # any name that does not prefix-match a cloud provider
reasoning_effort: "none"

Hermes's system prompt plus tool schemas is ~14k tokens, so a cold turn is 100-140 s to first token. Follow-up turns are cheap only if the KV checkpoint survives.

KV prefix reuse

The server keeps two live checkpoints: the state at the end of the last prompt and at the end of the last reply. A new request whose tokens start with the last reply's key prefills only the new tokens. Any other request in between overwrites both, and the next turn re-prefills everything (measured: 19.3k tokens, 138 s). While an agent is working, nothing else should use the port.

--prefix-reuse off|exact|auto controls it (exact is the default). The disk store (--kv-disk-dir <dir> --kv-disk-space-mb <n>) keeps many checkpoints keyed by token prefix; a 14k-token checkpoint is ~30 MB. Present but not yet validated on the Windows build.

The KV cache is stored 4-bit (FP4 values, E4M3 scales) regardless of the weight quant.

Gotchas

• Rebuilding: cargo build -p ds4-server compiles and then fails at link (no error[ lines); expected, link with hip\win\link_server.ps1. A running server keeps its own exe copy.
• Stopping: kill ds4-server, then wait for Windows commit to drop before starting again, or the loader thrashes at the commit limit.
• PowerShell 5.1 reads a BOM-less UTF-8 .ps1 as ANSI, which mangles the fullwidth bar in <｜...｜> tokens. Save scripts that contain them with a BOM.
• usage.prompt_tokens tells you whether a special token in your text was parsed as one id (5 with the chat template for a lone token) or split into text (18).

Runtime

Needs the patched ds4 / ds4-server runtime — llama.cpp, vLLM and Ollama cannot load the VQ blobs. Source, Windows/HIP build scripts, the launcher and the supervisor: staccs/ds4-winhip.

License

MIT, following the base model and the source abliteration.