Qwen3.6-35B-A3B-VQ-3.8bpw

15.7 GiB — smaller than the community 4-bit and a third closer to bf16.

A vector-quantized build of Qwen3.6-35B-A3B for Apple Silicon. Stock mlx-lm, no patches — the VQ runtime ships inside the checkpoint as model.py.

Changelog

2026-10-03 — MTP draft head

This repo now ships mtp-head-q6.safetensors, a 0.70 GiB multi-token-prediction draft head for speculative decoding. The weights and the bundled model.py are unchanged. The head is Qwen3.6-35B-A3B's own mtp.* layer from the bf16 release (stripped from the common MLX conversions), quantized to 6-bit, group 32. Knurlogic drafts with it automatically; see Speculative decoding. The exo --mtp note in the 2026-09 entry below is superseded: exo is no longer the serving path, and no head shipped until this revision.

2026-10-01 — runtime refresh; Knurlogic

The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac, drafting and vision instructions now use Knurlogic.

2026-09-17 — the full Qwen3.6-35B-A3B VQ ladder on one KL instrument

KL to this family's own bf16 teacher (cached top-64), 12288 tokens, on all three house corpora — prose, public code and literary — every rung scored on the same cache and the same positions, through a streamed pass validated against a direct full-model forward to all printed decimals.

Millinats per token; lower is closer to bf16. These rows are VQ rungs only and are not comparable to the affine-comparator table further down, which was measured prose-only at 2048 tokens on an earlier instrument.

Rank by KL, not perplexity. On this family perplexity is not monotone in quality: rungs that are measurably closer to the teacher can read a higher perplexity, because a damaged model can score better than bf16 on a finite sample. KL on cached teacher logits does not have that failure mode.

2026-09-09 — runtime refresh

Runtime refresh. The bundled model.py is updated so downloaders run exactly the code that was benchmarked; dense bundles now carry both runtimes.

• Faster prefill on affected geometries, measured per rung: a device-codebook kernel arm for large-codebook geometries (up to 1.46x on affected rungs), a ragged-subvector relaxation (up to 1.34x on affected rungs), a fused d8 arm, and a routing memo (≈2%). No blanket speedup is claimed across the lineup — gains apply only where the geometry engages the new paths.
• Output quality is unchanged: the kernel changes are bit-identical or 1-ULP-equivalent, and the routing memo is bit-identical (logits checksum verified).
• Speculative decoding (MTP): on repos that ship mtp-head-q6.safetensors, the sidecar works with the exo fork branch mtp-stage1 (github.com/noahzelezny/exo) — launch each node with exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves requests sequentially (the batch engine has no MTP path), so leave it off for concurrent / multi-agent workloads.

2026-09 — bundle refresh

Bundle refresh.

The bundled model.py runtime is updated as part of the 2026-09 refresh shared across the VQ lineup:

• Updated MoE expert kernels (uint32 code loads for the d2 geometry, template-free specialized dispatch). For the code geometries this model uses, outputs are verified equivalent to the previous bundle and the speed change is modest; the refresh is mainly for runtime consistency across the lineup.
• The bundle loads under both mlx-lm and mlx_vlm.
• Nothing you have downloaded breaks: weights are unchanged, only model.py is replaced, and the prior revision stays pinnable by commit hash.

Measured results

See also the 2026-09-17 entry in the changelog above, which places this rung against the rest of its family on the three-corpus KL instrument. The table below is the earlier prose-only measurement and is not comparable to it row-for-row.

Referee corpus, 2048-token windows, scored against the bf16 teacher on the same files with an unmodified mlx-lm. All sizes include the bf16 vision tower, which every build here carries:

33% less divergence than the 4-bit, at 3.3 GiB smaller. That is the comparison this build exists for.

Read the perplexity column carefully. At 4.7090 this build sits slightly below the bf16 teacher, and the 4-bit sits above it. That does not make it better than bf16 — perplexity is an aggregate over finite text and absorbs offsetting errors in both directions. KL measures distance to the teacher's distribution directly and does not. Treat anything within about 1% of the teacher as indistinguishable, and rank by KL.

Note the two metrics disagree in emphasis: KL and perplexity both favor this build over the 4-bit, while top-1 agreement is 89.55% against the 8-bit's 96.18%. For a 256-expert MoE that is expected — discrete routing flips swap one plausible token for another, which perplexity absorbs and argmax agreement punishes. Both numbers are honest; quote both.

Runtime

Speed on this revision (measured 2026-10-01)

One Mac Studio M3 Ultra, vqlab speed-pair: a fresh process per run, alternating with the reference, 3 pairs, 2048-token prompt, 128 generated tokens, no drafting.

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Comfortable on a 24 GB machine; workable on 16 GB at short context.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

With Knurlogic, which works out the settings and checks the model fits before loading it:

pip install knurlogic
hf download TheDrainFlorist/Qwen3.6-35B-A3B-VQ-3.8bpw --local-dir ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-3.8bpw
knurlogic serve ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-3.8bpw

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

Images work through Knurlogic directly; mlx-vlm is not needed.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships.

Speculative decoding (MTP)

Knurlogic drafts with mtp-head-q6.safetensors by default when it sits beside the weights (a full download includes it), and switches between drafting and plain decoding by measured cost.

• Turn it off: knurlogic serve <model> --set KNURLOGIC_MTP=off, or MTP Off in the load panel.
• Force it on, without automatic switching: --set KNURLOGIC_MTP_DYNAMIC=off.
• Requires Knurlogic 0.1.3 or newer (pip install -U knurlogic).

Measured on this build with this head (M4 Max, one Mac, greedy, thinking off, 400 tokens, 3 prompts; Knurlogic main + DSpark work, mlx 0.32.3):

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Without Knurlogic:

pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
  --model TheDrainFlorist/Qwen3.6-35B-A3B-VQ-3.8bpw \
  --prompt "Write a Python function that ..." \
  --max-tokens 512

Methodology

Uniform vector quantization of the experts at d=4, K=8192 — each 4-weight subvector stores one 13-bit index into a per-tensor 8192-entry fp16 codebook, with an fp16 scale per (row, 64 weights). Non-expert tensors are 8-bit. Experts are ≈92% of the model, so that is where the bytes are.

There is no depth schedule and no mixed allocation: every expert tensor uses the same geometry. Flat rungs are the reference points in this lineup because no mixed-allocation build we measured beat the flat rung at its own size.

Codebooks are fit in pure weight space — k-means over the weight subvectors, no Hessian, no activation statistics, no calibration corpus.

The fit is not seeded. k-means draws an unseeded subsample, so this artifact is reproducible in recipe and geometry but not bit-for-bit. Margins are therefore quoted against a measured fit-to-fit floor rather than against a repeated build; on this family that floor is 0.214 mnats, so the 25-mnat gap to the 4-bit is far outside it.

Verification

Every tensor was decoded from the published artifact and compared against the bf16 source; no tensor exceeds 3x the artifact's own median reconstruction error. The packed artifact reproduced its unpacked twin's KL score to every printed digit. The bundled runtime was exercised as the executing copy in a stock venv, not merely present in the folder.

Vision tower grafted in (333 tensors, 0.83 GiB); the text path is unchanged, verified against the same KL cache.

Limitations

• Top-1 agreement is 89.55% against the 8-bit's 96.18%. If your workload is sensitive to exact token choice rather than distributional quality, the 8-bit is measurably closer to bf16.
• No throughput measurement — see Runtime.
• No task-suite scores for this artifact.
• One asymmetry favors this build: the affine comparators quantize the MoE router, and this build leaves it at bf16. The bytes are trivial, but a router feeds an argmax over experts, so the effect is not bounded by the byte share. It is unmeasured.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: hello@thedrainflorist.com

Provenance

Base model: Qwen/Qwen3.6-35B-A3B — Apache-2.0. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.

Built with MLX and VQLab.