Qwen3.6-35B-A3B-VQ-4.6bpw

Within 0.5% of bf16 perplexity on two corpora, and smaller than the community 4-bit. 18.7 GiB (vision tower included): 0.991x on wikitext, 1.004x on code. The 8-bit holds 0.999x on both at 35 GiB; the 4-bit runs 1.041x / 1.046x at 19 GiB. Same harness, same files, every number. Runs on a 32 GB Mac.

A vector-quantized build of Qwen3.6-35B-A3B for Apple Silicon. Stock mlx-lm, no patches.

Changelog

2026-10-03 — MTP draft head

This repo now ships mtp-head-q6.safetensors, a 0.70 GiB multi-token-prediction draft head for speculative decoding. The weights and the bundled model.py are unchanged. The head is Qwen3.6-35B-A3B's own mtp.* layer from the bf16 release (stripped from the common MLX conversions), quantized to 6-bit, group 32. Knurlogic drafts with it automatically; see Speculative decoding. The exo --mtp note in the 2026-09 entry below is superseded: exo is no longer the serving path, and no head shipped until this revision.

2026-10-01 — runtime refresh; Knurlogic

The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac, drafting and vision instructions now use Knurlogic.

2026-09-17 — the full Qwen3.6-35B-A3B VQ ladder on one KL instrument

KL to this family's own bf16 teacher (cached top-64), 12288 tokens, on all three house corpora — prose, public code and literary — every rung scored on the same cache and the same positions, through a streamed pass validated against a direct full-model forward to all printed decimals.

Millinats per token; lower is closer to bf16. These rows are VQ rungs only and are not comparable to the affine-comparator table further down, which was measured prose-only at 2048 tokens on an earlier instrument.

Rank by KL, not perplexity. On this family perplexity is not monotone in quality: rungs that are measurably closer to the teacher can read a higher perplexity, because a damaged model can score better than bf16 on a finite sample. KL on cached teacher logits does not have that failure mode.

2026-09-09 — runtime refresh

Runtime refresh. The bundled model.py is updated so downloaders run exactly the code that was benchmarked; dense bundles now carry both runtimes.

• Faster prefill on affected geometries, measured per rung: a device-codebook kernel arm for large-codebook geometries (up to 1.46x on affected rungs), a ragged-subvector relaxation (up to 1.34x on affected rungs), a fused d8 arm, and a routing memo (≈2%). No blanket speedup is claimed across the lineup — gains apply only where the geometry engages the new paths.
• Output quality is unchanged: the kernel changes are bit-identical or 1-ULP-equivalent, and the routing memo is bit-identical (logits checksum verified).
• Speculative decoding (MTP): on repos that ship mtp-head-q6.safetensors, the sidecar works with the exo fork branch mtp-stage1 (github.com/noahzelezny/exo) — launch each node with exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves requests sequentially (the batch engine has no MTP path), so leave it off for concurrent / multi-agent workloads.

2026-09 — bundle refresh

Bundle refresh.

The bundled model.py runtime is updated as part of the 2026-09 refresh shared across the VQ lineup:

• Updated MoE expert kernels (uint32 code loads for the d2 geometry, template-free specialized dispatch). For the code geometries this model uses, outputs are verified equivalent to the previous bundle and the speed change is modest; the refresh is mainly for runtime consistency across the lineup.
• The bundle loads under both mlx-lm and mlx_vlm.
• Nothing you have downloaded breaks: weights are unchanged, only model.py is replaced, and the prior revision stays pinnable by commit hash.

Measured results

See also the 2026-09-17 entry in the changelog above, which places this rung against the rest of its family on the three-corpus KL instrument. The table below is the earlier prose-only measurement and is not comparable to it row-for-row.

Referee corpus, 2048-token windows, scored against the bf16 teacher on the same files with an unmodified mlx-lm:

At parity with bf16 on perplexity, at 26% of its size — and smaller than the 4-bit while being ≈5% better on ppl.

Read 0.991x as "at parity", not "better than bf16" — and here is the measurement that settles it. On a second corpus (code) the same build reads 1.004x: the sub-teacher reading does not reproduce, which is precisely what "corpus-specific" means. The KL column says the same thing on both corpora — this build sits 44.573 mnats (wiki) / 34.363 (code) from the teacher's distribution while the 8-bit sits at 7.449 / 4.708. Perplexity is an aggregate over finite text and absorbs offsetting errors in both directions; KL measures distance to the teacher directly and does not. Treat anything in 0.99-1.01x as indistinguishable from bf16, and read KL when you want to know which build is actually closer.

Perplexity is corpus-specific — compare only against models scored on the same files, never across harnesses.

Note the two metrics disagree in emphasis: ppl says parity, top-1 agreement says 90.75% vs the 8-bit's 96.18%. For a 256-expert MoE that is expected — discrete routing flips swap one plausible token for another, which perplexity absorbs and argmax-agreement punishes. Both numbers are honest; quote both.

The full sweep — every build we measured

Two codebook geometries were swept end to end. Every point below was scored on the same harness and corpus; nothing was discarded.

*Released sizes include the bf16 vision tower (+0.83 GiB); unreleased lab builds are text-only.

Reading notes: "rich layers 30-39" means those layers carry the d4·K2048 geometry and every other layer carries the cheap d2 geometry listed first. The placement control is this model's exact recipe mirrored end-for-end — same bytes, 0.028x worse — which is the measurement behind the layer schedule this release uses. The flat d2·K256 row is why top-1 agreement is reported but never used to rank builds: it has the best agreement and KL of any unreleased build and mid-pack perplexity.

Second corpus check (code)

The sweep above is scored on wikitext-style prose. VQ is known to behave differently by domain, so the released builds and both community quants were re-scored on a code corpus (MLX source), identical settings:

The ordering holds on both domains, and the margin over the 4-bit widens on code for both released builds. Every build also sits closer to the teacher on code than on prose — the domain is more predictable, not the quantization better.

Runtime

Speed on this revision (measured 2026-10-01)

One Mac Studio M3 Ultra, vqlab speed-pair: a fresh process per run, alternating with the reference, 3 pairs, 2048-token prompt, 128 generated tokens, no drafting.

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Earlier figures (previous runtime)

Single M3 Ultra, macOS, stock mlx-lm, GPU otherwise idle:

Measured on an M4 Max (128 GB), mlx-lm, 120-token greedy generation.

Comfortable on a 32 GB machine with context headroom — the 8-bit needs 48 GB+.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

With Knurlogic, which works out the settings and checks the model fits before loading it:

pip install knurlogic
hf download TheDrainFlorist/Qwen3.6-35B-A3B-VQ-4.6bpw --local-dir ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-4.6bpw
knurlogic serve ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-4.6bpw

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

Images work through Knurlogic directly; mlx-vlm is not needed.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships.

Speculative decoding (MTP)

Knurlogic drafts with mtp-head-q6.safetensors by default when it sits beside the weights (a full download includes it), and switches between drafting and plain decoding by measured cost.

• Turn it off: knurlogic serve <model> --set KNURLOGIC_MTP=off, or MTP Off in the load panel.
• Force it on, without automatic switching: --set KNURLOGIC_MTP_DYNAMIC=off.
• Requires Knurlogic 0.1.3 or newer (pip install -U knurlogic).

Measured on this build with this head (M4 Max, one Mac, greedy, thinking off, 400 tokens, 3 prompts; Knurlogic main + DSpark work, mlx 0.32.3):

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Without Knurlogic:

pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
  --model TheDrainFlorist/Qwen3.6-35B-A3B-VQ-4.6bpw \
  --prompt "Write a Python function that ..." \
  --max-tokens 512

Methodology

A depth schedule, not a uniform quantization. Layers 10-39 (the "tail") carry vector-quantized experts at d=2/K=512; layers 0-9 stay at d=4/K=2048. Non-expert tensors are 8-bit throughout.

The tail depth and the tail codebook were tuned separately, and that turned out to matter: a richer d2-K2048 tail produced a larger and worse artifact (20.7 GiB, 1.000x) than this one. The tail had been sized while the geometry was d4 and nobody re-tuned it afterward. Going cheaper again (d2-K256) overshot — 16.5 GiB but 1.002x — so K512 is the knee, found by bracketing rather than by extrapolation.

Verification

Every tensor was decoded from the published artifact and compared against the bf16 source; no tensor exceeds 3x the artifact's own median reconstruction error. Packing was separately verified as a pure representation change: the packed model scores identically to the unpacked one to three decimals.

Limitations

• Top-1 agreement is 90.75% against the 8-bit's 96.18%. If your workload is sensitive to exact token choice rather than distributional quality, the 8-bit is measurably closer to bf16.
• Perplexity was measured on one corpus. A code-heavy or non-English workload may rank these builds differently.
• No blind human-preference evaluation was run on this model (unlike our gemma-4 releases, which needed one); the claim here rests on perplexity, which is valid for this family.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: hello@thedrainflorist.com

Provenance

Base model: Qwen/Qwen3.6-35B-A3B — Apache-2.0. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.

Built with MLX and VQLab.