DeepSeek-V4-Flash-0731 GGUF — Quantized by BatiAI
The 2026-07-31 refresh of DeepSeek-V4-Flash — quantized from the official weights, verified in Korean. 284B-A13B MoE with CSA+HCA hybrid attention. Runs on a single high-RAM Mac.
📦 Quantizations
| Quant | Size | Shards | Best for |
|---|---|---|---|
| Q3_K_M | 135 GB | 4 | M4 Max 128GB (tight) → 192GB Mac Studio |
| Q4_K_M | 172 GB | 4 | M2 Ultra 192GB / server |
Built directly from official DeepSeek weights via a Q8_0 intermediate, BatiAI‑signed
(general.author: BatiAI). Q5+ intentionally omitted — the size/benefit tradeoff doesn't
land on any real Mac tier.
🆕 What changed in 0731. Same architecture as the original V4‑Flash (43 layers, 256 experts, vocab 129280) with refreshed weights — and a new storage format: experts are stored INT8 with UE8M0 scales (
scale_fmt: ue8m0) instead of plain FP8. That is why these files are slightly larger than our earlier V4‑Flash GGUFs, and why conversion needstorch>=2.7(E8M0 dtype).
✅ Verified (this build, Q3_K_M)
Captured greedy runs — not copied from the base model card:
| Test | Output |
|---|---|
| Math | 127+58= → 185 |
| Korean | "서울은 대한민국의 수도로, 역사와 현대가 공존하는 역동적인 도시입니다. / 불고기는 달콤한 간장 양념에 재운 소고기를 불에 구워내는… / 비빔밥은 밥 위에 다양한 나물과 고기, 고추장을 얹어… / 김치는 배추를 소금에 절여 고춧가루와 젓갈 등으로 양념해 발효시킨…" — fluent, zero token mixing or loops |
| Tool call | {"tool":"get_weather","args":{"city":"부산"}} — exact JSON |
🚀 Usage
⚙️ Runtime: batiai/bati.cpp — DeepSeek‑V4's CSA+HCA hybrid attention (
deepseek4) is not in mainline llama.cpp. Build our fork.
hf download batiai/DeepSeek-V4-Flash-0731-GGUF "DeepSeek-V4-Flash-0731-Q3_K_M-*.gguf" --local-dir ./v4f
./llama-cli -m ./v4f/DeepSeek-V4-Flash-0731-Q3_K_M-00001-of-00004.gguf -ngl 99 -c 8192
⚠️ Two things the base repo doesn't tell you (we hit both)
- No chat template ships with this model — neither the original nor the GGUF has one.
Use DeepSeek's format explicitly:
<｜User｜>your question here<｜Assistant｜> - CUDA offload used to abort in
ggml_cuda_op_concat(GGML_ASSERT(src0->type == GGML_TYPE_F32)) because the backend advertised concat support for types its kernel can't handle. Fixed in bati.cpp — the CUDA backend now reports concat support only for F32, so those nodes fall back to CPU instead of crashing. Use bati.cpp at or after that fix, or run CPU/Metal. (Apple Metal users were never affected.)
✨ What BatiAI did
- Quantized from the official DeepSeek weights — never a re‑quant of someone else's GGUF
- Handled the new UE8M0 / INT8‑expert storage format end to end
- Verified in Korean — most GGUF publishers never check this
- Fixed the runtime bug we found along the way, in the open, in bati.cpp
- BatiAI metadata‑signed
📜 License — MIT
Base model © DeepSeek, MIT. Quantized weights redistributed under the same terms.
Who we are. BatiAI builds on‑device Korean AI — BatiFlow runs LLMs, speech‑to‑text (batisay), document OCR (batisee) and speaker diarization locally on a Mac. No audio, no documents, no prompts leave the device. Browse the whole line at huggingface.co/batiai.