Qwen3.5-397B-A17B-VQ-3.5bpw

141.2 GiB text weights — the quality build. Also ships the bf16 vision tower (+0.85 GiB) and an optional MTP draft head (+5.41 GiB); full download 147.4 GiB.

A vector-quantized build of Qwen3.5-397B-A17B for machines with memory to spend: the strongest quantization we know how to make of this model at this size, on stock mlx-lm, no patches.

Changelog

2026-10-02 — First release

A v2 (mixed-codebook) build derived from VQ-3.1bpw: the routed-expert projections (gate/up/down) of layers 42-59, the back 18 of 60 layers, are refitted at d2/K256 from the bf16 teacher, and layers 0-41 keep the 3.1's bytes exactly. vq-skipzero is applied to all 138 eligible modules.

The 3.1's own card carries that build's history; it is not repeated here.

Measured results

KL to this family's own bf16 teacher (cached top-64), 12288 tokens per corpus, on all three house corpora — prose, public code and literary — every row scored on the same cache and the same positions.

Millinats per token; lower is closer to bf16. The 2.2-2.6 and spicyneuron 2.6bit rows are from the 2026-09-17 ladder; the 3.1, 3.5 and spicyneuron 3.5bit rows were scored on the same cache on 2026-10-02.

Paired deltas on identical positions. Against the 3.1 this build is better on all three corpora: prose -32.5 (t -18.6), code -8.4 (t -9.2), literary -11.5 (t -7.6). Against spicyneuron 3.5bit, ours is lower on prose (t 13.2) and literary (t 7.9) at 24.4 GiB less text. On code, 24.9 against 28.6 is not a significant difference (t 2.0; our gate needs |t| > 2), so we report it as a tie.

Rank by KL, not perplexity. On this family perplexity is not monotone in quality: rungs that are measurably closer to the teacher can read a higher perplexity, because a damaged model can score better than bf16 on a finite sample. KL on cached teacher logits does not have that failure mode.

Task benchmarks

Task benchmarks have not been run on this build.

Hardware

141.2 GiB text weights (147.4 GiB full download incl. tower + MTP head). mlx-lm loads only the text weights, so resident ≈141.2 GiB — this does not fit a 128 GB machine. You need either:

• a single Apple Silicon machine with ≥ 192 GB unified memory, or
• two or more Macs with Knurlogic (e.g. 96 GB + 128 GB over Thunderbolt; see Run it). This build was verified on a two-Mac pipeline split.

VQ codebooks must be replicated across machines, not sliced: a runtime that slices them produces fluent nonsense rather than an error. This artifact's bundled model.py carries a runtime guard that raises loudly if a sliced codebook reaches the forward path, so a misconfigured cluster fails instead of lying.

Speed

Speed for this revision has not been measured yet; no number is claimed here.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

With Knurlogic, which works out the settings and checks the model fits before loading it:

pip install knurlogic
hf download TheDrainFlorist/Qwen3.5-397B-A17B-VQ-3.5bpw --local-dir ~/Knurlogic/Models/Qwen3.5-397B-A17B-VQ-3.5bpw
knurlogic serve ~/Knurlogic/Models/Qwen3.5-397B-A17B-VQ-3.5bpw

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

At 147.4 GiB on disk this needs a Mac with more memory than that, or two Macs: install the same Knurlogic version and the model on each, run knurlogic ui --host cluster on both, and launch it from the page with both machines selected. Knurlogic splits the layers between them.

Images work through Knurlogic directly; mlx-vlm is not needed.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships.

Without Knurlogic:

pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
  --model TheDrainFlorist/Qwen3.5-397B-A17B-VQ-3.5bpw \
  --prompt "Explain vector quantization briefly." \
  --max-tokens 1000

Speculative decoding (MTP)

mtp-head-q6.safetensors (5.41 GiB) is the model's own MTP draft head. Cluster speculative decoding is live: Knurlogic loads the shipped head and drafts automatically, on one Mac or split across several, with nothing to enable (outputs exactly the base model's via rejection sampling). Measured on other rungs of this family, the head drafts well, but this family's stock pipeline decode does not degrade with generation length, so there is little for speculation to recover. Expect parity; per-rung numbers for this artifact are not yet measured.

Vision

The artifact includes the full 333-tensor vision tower at source precision (0.85 GiB). mlx-lm is text-only for this architecture and ignores it; Knurlogic loads it from this folder directly, without mlx-vlm.

The sizes quoted above are the download: they include this tower. Because mlx-lm does not load it, resident memory runs ≈0.85 GiB below the disk figure.

Methodology

Mixed codebooks by position in the network. This is a v2 (mixed-codebook) build. Layers 0-41 use the 3.1bpw's uniform geometry, d4/K2048: each 4-weight subvector stores one 11-bit index into a per-tensor 2048-entry fp16 codebook. Layers 42-59 — the routed-expert projections (gate/up/down) of the back 18 of 60 layers — are refitted at d2/K256 from the bf16 teacher: one 8-bit index per 2 weights, which is 4.0 bits/weight for codes against 2.75 in the front. Layers 0-41 keep the 3.1's bytes exactly. The allocation follows the position law: early layers tolerate cheap bits, and enrichment pays only in the back of the network.

vq-skipzero. Some output rows in the teacher have weights around 1e-29. vq-skipzero drops those rows' codes and scales on disk and in memory; they output exact zeros. It is applied to all 138 eligible modules; the live rows are byte-identical (vqlab sz-check PASS).

Mixed precision by layer sensitivity. Attention, MoE routers, embeddings, and the output head stay at higher precision — they are a small fraction of the parameters but errors there propagate through every token. The MoE experts are ≈85% of the model and individually far more tolerant, so they absorb the aggressive quantization.

Vector quantization instead of scalar rounding. Instead of giving each weight a rigid set of levels, this build learns a codebook of joint patterns over small weight groups and stores one index per group. The codebook's entries sit where the weight distribution actually is, rather than on a uniform lattice — which is why this beats scalar quantization at matched size rather than merely matching it. Per-tensor codebooks, with an fp16 scale per (row, 64 weights).

Codebooks are fit in pure weight space — k-means over the weight subvectors, no Hessian, no activation statistics, no calibration corpus. That is a deliberate choice: calibration-fitted methods we tested (GPTQ- and DWQ-style) reduced layer error while making end-to-end perplexity worse on this architecture, and they bias the result toward whatever text the calibration set contains. Weight-space fitting has no such domain preference.

Sub-byte bit-packing. Codes are packed into uint32 words (row-local, 32-code blocks) rather than padded to whole bytes, which is what makes the non-byte-aligned sizes possible at all. Packing is a pure representation change.

How it was evaluated. KL to the bf16 teacher on cached top-64 logits, 12288 tokens per corpus, on three house corpora (prose, code, literary), with paired comparisons on identical positions. KL is the ranking instrument; see the table above.

Verification

Release gates passed on this artifact before upload: check-bundle PASS, and generation verified through the shipping runtime across two Macs (pipeline split): a 2667-token generation (1379 thinking tokens) finished on its own stop token with no repeated 8-word span. The vq-skipzero live rows were checked byte-identical with vqlab sz-check (PASS).

Limitations

• Needs ≥ 192 GB unified memory or a cluster — see Hardware above.
• This is a thinking model (Qwen3.5 family): it spends tokens reasoning before answering. Budget max_tokens accordingly.
• Speed and task benchmarks have not been measured for this build.

Acknowledgment

spicyneuron's 397B quants are what made this model runnable on my hardware in the first place — they were the artifacts that fit when nothing else did, and they were the reference this work was measured against throughout. This release is offered in that same spirit: the full method, the experiments that failed as well as the ones that worked, and comparator numbers re-measured on one harness so the claims can be checked rather than taken on trust.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: hello@thedrainflorist.com

Provenance

Base model: Qwen/Qwen3.5-397B-A17B — Apache-2.0. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Derived from VQ-3.1bpw with layers 42-59 refitted from the bf16 teacher. Quantization: TheDrainFlorist, 2026.

Built with MLX and VQLab.