DeepSeek-V4-Flash-VQ-3.2bpw

106.4 GiB text weights, 3.21 bits per weight. Fits one 128 GB Apple Silicon Mac. Also ships an MTP draft head (+2.38 GiB); full download 108.8 GiB. Text only: no vision tower.

A vector-quantized build of DeepSeek-V4-Flash. The official release (FP4 experts, FP8 attention) is about 149 GiB; this is about 28% smaller.

Runtime note. This architecture (deepseek_v4) is not in stock mlx-lm (upstream support is still in open pull requests), and plain transformers does not register it yet. Stock mlx-lm and plain transformers cannot load this model. It runs on Knurlogic, which ships the deepseek_v4 architecture; use Knurlogic 0.1.3 or newer. See Run it.

Changelog

2026-10-04 — rescored on Knurlogic 0.1.3

Rescored on Knurlogic 0.1.3, which matches DeepSeek's reference numerics (router scores, logits and compressor pooling in float32, single rounding in weighted norms); this model now requires Knurlogic 0.1.3 or newer. The weights are unchanged, and the paired verdict against uniform d4/K2048 is unchanged.

2026-10-02 — First release

A v2 (mixed-codebook) build. Every layer's routed experts are d4/K2048; layers 24-35 are d4/K4096. Attention, shared expert, router, embeddings and head are 8-bit affine.

Measured results

KL to the exact teacher, cached top-64, 12288 tokens per corpus, on all three house corpora (prose, code, literary), every row scored on the same cache and the same positions.

There is no bf16 DeepSeek-V4-Flash: the official release's FP4 experts are the model. The teacher here is that release converted to MLX layout with the experts bit-identical (mxfp4) and the FP8 attention dequantized exactly to bf16. Teacher caches were rebuilt and reproduced bit-for-bit.

Scored with DeepSeek's reference numerics. DeepSeek's own inference code clamps the SwiGLU of the shared expert as well as the routed experts; mlx-lm's deepseek_v4 leaves the shared expert unclamped. Every row below is scored with the clamp (teacher and builds alike). The clamp fires on about 0.03% of shared-expert activations, and the teacher predicts better with it on all three corpora, so it is the model's real numerics. This table replaces an earlier one measured without the clamp: every row moved by under 4%, and the ordering and the paired verdict are unchanged. Knurlogic applies the clamp from 0.1.3.

The VQ-3.2bpw row was re-measured 2026-10-04 on Knurlogic 0.1.3; the other three rows are from the earlier runtime and would differ by a few millinats.

Millinats per token; lower is closer to the teacher. The mlx-community row shares this build's FP4 experts, so it is the floor that the 8-bit-attention skeleton inherits: at most about 7 millinats on any corpus here.

Paired deltas on identical positions (earlier runtime). Against uniform d4/K2048, this build is better on all three corpora: prose t -8.0, code t -5.4, literary t -5.4, for +2.3 GiB.

How the K4096 band was chosen. A leave-one-out map (each 12-layer band of a uniform build restored to the exact experts in turn) put the most damage in layers 24-35, so those layers get the larger codebook. Two intermediate K256-plus-K2048 builds from that search are not re-scored here: they were assembled with an older runtime file and cannot be rebuilt byte-for-byte.

Teacher perplexity on these corpora: prose 2.329, code 1.560, literary 1.099 (the literary corpus is public-domain text the model has largely memorized).

Rank by KL, not perplexity. KL on cached teacher logits is the ranking instrument; perplexity aggregates and can absorb offsetting errors.

Task benchmarks

Task benchmarks have not been run on this build.

Hardware

106.4 GiB text weights; resident is about the same, plus 2.38 GiB when the MTP draft head is bound. This fits a single Apple Silicon machine with 128 GB unified memory. It also runs split across two Macs with Knurlogic's tensor split (about 54 GiB per Mac), over TCP or RDMA (Thunderbolt 5).

Speed

No absolute tokens-per-second figure is claimed. Measured as a same-session ratio on one M4 Max (128 GB), greedy, 800 tokens, three runs per arm: drafting decodes about 1.42x faster than plain decoding, with an acceptance rate of 0.95 (measured with the exact head; the shipped 6-bit head accepts at the same rate).

Speculative decoding (MTP)

mtp-head-q6.safetensors is the official release's multi-token-prediction layer (mtp.0). Its routed experts are vector-quantized at the trunk's own geometry (d4/K2048, 11-bit codes); its dense projections (attention, e_proj/h_proj, shared expert) are 6-bit affine (group 64); router, norms and hyper-connections are exact. Knurlogic finds it beside the weights and drafts with it by default; drafting preserves the output distribution but is not bit-identical to plain decoding. Measured against an exact head (FP4 experts, bf16 dense, 3.50 GiB) and two intermediate heads on the same prompts: acceptance was the same for all of them (0.95 on one Mac, 0.885 across a two-Mac tensor split), and no speed difference between heads was resolvable, so the smallest head ships.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

Stock mlx-lm and plain transformers cannot load this model (see the runtime note above). Use Knurlogic, which ships the deepseek_v4 architecture, works out the settings and checks the model fits before loading it:

pip install "knurlogic>=0.1.3"
hf download TheDrainFlorist/DeepSeek-V4-Flash-VQ-3.2bpw --local-dir ~/Knurlogic/Models/DeepSeek-V4-Flash-VQ-3.2bpw
knurlogic serve ~/Knurlogic/Models/DeepSeek-V4-Flash-VQ-3.2bpw

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships. The repo also ships Knurlogic's deepseek_v4 chat template as chat_template.jinja.

Methodology

Mixed codebooks by position in the network. This is a v2 (mixed-codebook) build. The quantized tensors are the routed experts: 256 per layer across 43 layers, gate/up/down, 129 modules, about 277B of the 284B parameters. Every layer is d4/K2048: each 4-weight subvector stores one 11-bit index into a per-tensor 2048-entry codebook. Layers 24-35 are d4/K4096, 12-bit codes. Attention, shared expert, router, embeddings and head are 8-bit affine, the skeleton of the mlx-community MLX conversion, whose experts were verified byte-identical to the official release. Overall 3.21 bits per weight over the 284.3B main-model parameters.

The allocation was measured on this family, not inherited. A leave-one-out band test on a uniform d4/K256 build showed DeepSeek's damage peaks in layers 24-35 and is lowest in the last 7 layers. That is unlike Qwen, where the back of the network is where bits pay. Restoring layers 24-35 alone removed 287 of 631 prose and 328 of 416 literary millinats, so layers 24-35 got the extra bits.

Vector quantization instead of scalar rounding. Instead of giving each weight a rigid set of levels, this build learns a codebook of joint patterns over small weight groups and stores one index per group. The codebook's entries sit where the weight distribution actually is, rather than on a uniform lattice.

Codebooks are fit in pure weight space: k-means over the weight subvectors, no Hessian, no activation statistics, no calibration corpus. Weight-space fitting has no domain preference.

Sub-byte bit-packing. Codes are packed to their true bit width rather than padded to whole bytes, which is what makes the non-byte-aligned sizes possible at all. Packing is a pure representation change.

How it was evaluated. KL to the exact teacher on cached top-64 logits, 12288 tokens per corpus, on three house corpora (prose, code, literary), with paired comparisons on identical positions. See the table above.

Verification

Generated through Knurlogic on one 128 GB Mac: the 106.4 GiB loaded in 38.7 s, and a greedy 1772-token answer (748 thinking tokens) finished on its own stop token with no repeated 8-word span. Across two Macs with Knurlogic's tensor split (TCP and RDMA, drafting on and off), 1500-token greedy generations stayed coherent with no repeated 8-word span; with drafting on, 664 of 750 drafts were accepted. check-release PASS (the tokenizer round-trips once Knurlogic registers the model type).

Limitations

• Needs a runtime that ships the deepseek_v4 architecture (Knurlogic 0.1.3 or newer). Stock mlx-lm and plain transformers cannot load it.
• The 8-bit attention skeleton accounts for at most about 7 millinats, per the mlx-community comparator row.
• KL was measured at the scoring path (prefill). Long generation was verified by smoke only.
• This is a thinking model: it spends tokens reasoning before answering. Budget max_tokens accordingly.
• No absolute speed figure and no task benchmarks are published for this build.

Acknowledgment

This build stands on two community efforts: the mlx-community MLX conversion of DeepSeek-V4-Flash, whose attention and embedding skeleton it reuses (its experts were verified byte-identical to the official release), and the community port of the deepseek_v4 architecture to MLX that Knurlogic vendors to serve it. The release is offered in the same spirit: the full method, the experiments that failed as well as the ones that worked, and every comparison re-measured on one harness so the claims can be checked rather than taken on trust.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: hello@thedrainflorist.com

Provenance

Base model: deepseek-ai/DeepSeek-V4-Flash — MIT. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Fitted with VQLab (fit-moe --family deepseek_v4, seed 1234) and packed to true bit width; the build record is vqlab_provenance.json in this repo. Quantization: TheDrainFlorist, 2026.

Built with MLX and VQLab.