DeepSeek V4.1 Flash GGUF for DwarfStar

DeepSeek V4.1 Flash 1017

The 1017 Q2 and matching BF16 vision encoder come from deepseek-ai/DeepSeek-V4.1-Flash-1017. They use the DwarfStar GGUF format. Pair the 1017 encoder with the 1017 language model; the original release remains available below.

hf download kyuz0/deepseek-v4.1-flash-gguf DeepSeek-V4.1-Flash-1017-Q2.gguf --local-dir gguf
hf download kyuz0/deepseek-v4.1-flash-gguf DeepSeek-V4.1-Flash-1017-vision-BF16.gguf --local-dir gguf

Tested ROCm runtime

The 1017 files were tested on AMD gfx1151 with ROCm 10.0, using kyuz0/ds4 with these source overlays in order:

1. Bounded decoder replay.
2. Raw completion prompts.
3. Loaded-model modalities.

The combined source tree is 57e71ca100ac7080716eb56e767263bca6983544. For 1017 vision, apply the included checkpoint compatibility patch to that tree before building. It selects the matching encoder revision from the language checkpoint metadata. The image preprocessing and inference equations are unchanged.

A short text invocation with the tested options is:

./ds4 --rocm --quality \
  --model gguf/DeepSeek-V4.1-Flash-1017-Q2.gguf \
  --ctx 4096 --tokens 128 --threads 4 --nothink \
  --temp 1 --top-p 0.95 --min-p 0 --seed 1017 \
  --prompt 'Calculate 17 * 19. Give the result, then explain your calculation in two short sentences.'

Text generation and the public multimodal session API were exercised with these weights. The vision API comparison covers 32 cases and 40 image occurrences with the matching 1017 encoder. The reported distribution comparisons use the runtime's quality mode.

1017 DSpark support

DeepSeek-V4.1-Flash-1017-DSpark-MXFP4.gguf is the matching external draft model for the 1017 checkpoint: 7,966,294,016 bytes (7.42 GiB), 81 tensors. It contains the three official DSpark stages from the same pinned source revision as the 1017 Q2, with native MXFP4 expert values preserved and Q8/F16/F32 for the other tensors. No additional imatrix is required.

hf download kyuz0/deepseek-v4.1-flash-gguf DeepSeek-V4.1-Flash-1017-DSpark-MXFP4.gguf --local-dir gguf

DS4_ENABLE_V41_DECODER_SWA_BOUNDED_REPLAY=1 ./ds4-server --rocm \
  --model gguf/DeepSeek-V4.1-Flash-1017-Q2.gguf \
  --mtp-model gguf/DeepSeek-V4.1-Flash-1017-DSpark-MXFP4.gguf \
  --dspark --ctx 262144 --host 127.0.0.1 --port 8000

Use the tested ROCm runtime described above. For DSpark serving, omit --batched-session, including --batched-session 1; native session batching disables speculative decoding in this runtime. SSD expert streaming cannot be combined with DSpark. Leave confidence on its runtime default; the five-token draft cap and adaptive scheduling are supplied by the model and engine.

Standalone ROCm checks passed for CLI and HTTP generation/EOS, a generated Python function on five executable cases, and a tool-call round trip. Actual draft proposals and acceptance were verified with zero reported DSpark errors. These focused checks do not establish long-context or distributed DSpark quality.

1017 quantization and calibration

Q2 follows the public antirez recipe: IQ2_XXS routed gate/up experts, Q2_K routed down experts, Q8_0 attention/shared/output weights, and F16/F32 tensors elsewhere. Native Engram data is retained losslessly and read from the GGUF on disk. The separate vision encoder retains the source BF16 tensors.

The tested resident runtime prepared 151.76 GiB of language-model tensor spans, before context and runtime buffers. The file also contains 188.83 GiB of Engram data. Keep it on a fast local SSD and allow additional memory for the selected context and vision encoder.

The importance matrix was collected from the native 1017 checkpoint using 49,152 text tokens and 6,246 vision-conditioned prefill tokens from 7 images. Of 15,360 routed experts, 15,268 were observed; 92 use the public converter fallback. This is a new 1017 calibration matrix. The Q2 was converted directly from the original native weights using this matrix.

The reusable 1017 importance matrix is available with its calibration provenance and pinned Q2 conversion instructions. It is the exact 770,707,978-byte matrix used for this release; no new calibration run is needed to reuse it.

The native QA references and scorer instructions are public. The routine comparison uses 100 cases; vision, long continuations and deep-context panels are optional downloads.

1017 paired quality measurements

These comparisons teacher-force identical frozen token histories through the original native 1017 checkpoint and Q2. The original weights are mixed FP8/FP4. Forward KL uses full-vocabulary distributions at temperature 1. Results include quantization and runtime arithmetic; image results include the complete existing image pipeline.

NLL has the same definition as the original DwarfStar scorer’s avg_nll: average negative log probability of the reference continuation tokens. Lower is better on the same frozen targets; positive ΔNLL means Q2 assigns them less probability. The short panel reuses the V4.1 Flash September 19 continuations with official 1017 token IDs and newly collected 1017 native logits. Scores from different checkpoints, tokenizers or panels are not interchangeable. Top-token agreement is not answer accuracy.

Deep coverage is partial: six lookup cases at 4K, 8K, 16K, 32K, 64K and 128K, plus one 4K multihop case. The remaining 29 cases are untested. Of the 2,688 scored positions, 127 are teacher-forced answer tokens and the rest are prompt probes. Answer-only forward KL is 0.024531, with matching top choices at all 127 positions. This does not establish independently generated answer accuracy or coverage of the other deep task types.

Expanded text groups

Groups have different token denominators. The multilingual group contains short frontiers rather than full-answer accuracy tests. Negative ΔNLL on fixed targets does not establish better task performance.

Natural image-aligner outputs were compared before prompt-sentinel insertion: 32 cases, 40 image occurrences, mean row cosine 0.998737 and RMSE 0.001771 against native BF16 features.

The generation-history panels score saved native continuations, not independently generated Q2 task success. All four long native traces reached their 4,096-token caps. The pilot and prefill control share a history. There is no combined quality percentage, universal acceptance threshold or subtraction of native numerical controls as an error floor.

Source revision: 1bae0e1088993b9b82de08f59de5be5a34e42170. The included upstream MIT license applies.

Original V4.1 Flash release

Calibrated weights for the Metal implementation in DwarfStar, developed on the ds4.1flash branch. These are not interchangeable with DeepSeek V4 Flash weights or its DSpark drafter.

The language GGUFs each include 188.83 GiB of native FP8 Engram tables. DwarfStar reads the needed rows directly from disk; it never makes the whole table resident. Keep the GGUF on a fast local SSD. The main-weight sizes above do not include context or runtime buffers.

Download and Run

From a DwarfStar checkout with V4.1 support:

./download_model.sh ds41f-q2
./ds4 -m gguf/DeepSeek-V4.1-Flash-Q2.gguf --ssd-streaming --ctx 32768

Use SSD streaming on a single 128 GB Mac. Q2 also fits resident tensor parallelism across two 128 GB Macs with RDMA, or full residency on a larger Mac. Q4 needs SSD streaming on smaller Macs; the resident target is a 512 GB Mac. It does not fit resident TP across two 128 GB Macs.

./download_model.sh ds41f-q4
./download_model.sh ds41f-vision

Q4 exceeds Hugging Face's single-file limit, so it is stored in two binary parts. The downloader joins them into DeepSeek-V4.1-Flash-Q4.gguf, verifies the complete file and removes the temporary parts. Allow another 37 GiB of free disk space while joining. Rerun the command after an interruption. The individual parts are not runnable GGUFs.

For images, add --vision gguf/DeepSeek-V4.1-Flash-Vision.gguf to the language model command. The same model and memory options work with ds4-agent and ds4-server.

Quantization

Q2 uses IQ2_XXS routed gate/up experts and Q2_K down experts. Q4 uses Q4_K for all routed experts. Both retain Q8 attention projections, shared experts and output, with F16/F32 tensors elsewhere and unchanged native Engram data. They use the same 8,192-token activation imatrix, not a requantization of one GGUF into the other.

Source: deepseek-ai/DeepSeek-V4.1-Flash at df42c109f1defefcbfcedbe7d905718a12266e40. The separate encoder comes from the same checkpoint. See the included upstream MIT license.