DeepSeek V4 Flash, GGUF
DeepSeek V4 Flash is a large-scale Mixture-of-Experts model optimized for ultra-long context reasoning and efficient inference. It is built with hybrid CSA + HCA attention and manifold-constrained hyper-connections. With FP4 and FP8 mixed precision and strong agentic tool-calling capabilities, it is designed for scalable, high-efficiency reasoning and multi-domain workloads.
This repository holds Vultr's GGUF quantizations of it. They are hybrid quantizations: rather than applying one precision to the whole network, each layer's precision is chosen from a measured importance score, so the bits go where they change the output most.
Model Overview
| Attribute | Value |
|---|---|
| Base model | deepseek-ai/DeepSeek-V4-Flash |
| Architecture | deepseek4, Mixture-of-Experts |
| Parameters | 284B total |
| Active parameters | approximately 13B |
| Layers | 43 |
| Hidden size | 4,096 |
| Attention heads | 64 |
| Experts | 256 routed, 6 selected per token |
| Context length | up to 1M tokens |
| Vocabulary | 129,280 |
| License | MIT |
| Attribute | Value |
|---|---|
| Base model | deepseek-ai/DeepSeek-V4-Flash |
| Architecture | deepseek4, Mixture-of-Experts |
| Parameters | 284B total |
| Active parameters | approximately 13B |
| Layers | 43 |
| Hidden size | 4,096 |
| Attention heads | 64 |
| Experts | 256 routed, 6 selected per token |
| Context length | up to 1M tokens |
| Vocabulary | 129,280 |
| License | MIT |
Quantizations
| File | Type | Size | KLD vs original |
|---|---|---|---|
| DeepSeek-V4-Flash-MXFP4-*.gguf (2 parts) | MXFP4 | 154.99 GB | 0.0093 |
| DeepSeek-V4-Flash-IQ3_M.gguf | IQ3_M | 136.11 GB | 0.0864 |
| DeepSeek-V4-Flash-IQ3_S.gguf | IQ3_S | 134.68 GB | 0.0876 |
| File | Type | Size | KLD vs original |
|---|---|---|---|
| DeepSeek-V4-Flash-MXFP4-*.gguf (2 parts) | MXFP4 | 154.99 GB | 0.0093 |
| DeepSeek-V4-Flash-IQ3_M.gguf | IQ3_M | 136.11 GB | 0.0864 |
| DeepSeek-V4-Flash-IQ3_S.gguf | IQ3_S | 134.68 GB | 0.0876 |
KL divergence measures how far a quantized model's next-token probabilities drift from those of the original model as released (its routed experts are already in 4-bit MXFP4), averaged over every token of the evaluation text. The lower the value, the closer the quantized model is to the original; zero would mean identical output.
Evaluation
Every variant in this repository was evaluated against the original model as released (its routed experts are already in 4-bit MXFP4) for perplexity, KL divergence and top-token agreement, under identical conditions. The full results are maintained on the model's page in the Vultr Model Library:
https://docs.vultr.com/models/deepseek-v4/deepseek-v4-flash?tab=quantized
That page is the reference for these files. The summary above is an excerpt from it.
HellaSwag and WinoGrande are not included: llama.cpp cannot yet score multiple-choice tasks on this architecture.
Run
llama-server -hf vultr/DeepSeek-V4-Flash-GGUF:MXFP4
The model's chat template is embedded in every file, so no extra prompt formatting is needed.
The repository also includes mtp-DeepSeek-V4-Flash-Q8_0.gguf, the model's multi-token prediction (MTP) head, for optional speculative decoding with any quant above:
llama-server -hf vultr/DeepSeek-V4-Flash-GGUF:MXFP4 --spec-type draft-mtp
License
These files are derived from deepseek-ai/DeepSeek-V4-Flash and are distributed under the same license, MIT. The model is the work of DeepSeek; Vultr produced the quantizations.