Qwen3.6-35B-A3B — GGUF with a 333 MB MTP head
The current model is Qwen3.6-35B-A3B-IQ2XXS-Q2K-Q8.gguf, built for QuarkStar.
It combines the Antirez asymmetric expert quantization with an integrated 332.78 MB MTP head. All 733 target-model tensor payloads are unchanged from the original Antirez IQ2_XXS/Q2_K/Q8-rest model. The complete file is 12.07 GB (11.24 GiB) and was tested fully resident on a 16 GB AMD BC-250.
All older model files in this repository are deprecated. Use the integrated file above for new installations.
llama.cpp mainline
Both the Q2 model above and Qwen3.6-35B-A3B-IQ4XS-Q4K-Q8.gguf work with llama.cpp mainline. QuarkStar currently supports only the Q2 model; the Q4 model does not work with QuarkStar.
Download and run
Use an updated QuarkStar checkout with the q2 download target and optimized MoE MTP support:
git clone https://github.com/Ninnix/q36
cd q36
./download_model.sh q2
make
./q36 --mtp --temp 0 --nothink -p "Write a C function that reverses a linked list."
Authentication is optional for this public repository:
./download_model.sh q2 --token TOKEN
To select the file explicitly or start the server:
./q36 -m gguf/Qwen3.6-35B-A3B-IQ2XXS-Q2K-Q8.gguf --mtp --temp 0 -p "Hello"
./q36-server -m gguf/Qwen3.6-35B-A3B-IQ2XXS-Q2K-Q8.gguf --mtp --ctx 32768
--mtp selects the embedded head. Omitting it excludes the extra block from the resident GPU weight cache. The downloader resumes partial transfers and updates ./q36moe.gguf to the selected file. A separate MTP download is no longer needed.
Quantization recipe
| Tensor class | Target blocks 0–39 | MTP block 40 |
|---|---|---|
| Routed gate/up experts | IQ2_XXS | IQ2_XXS |
| Routed down experts | Q2_K | Q4_K |
| Attention and other projections | Q8_0 | Q8_0 |
| Shared experts | Q8_0 | Q8_0 |
| Routers and norms | F32 | F32 |
| Token embeddings and output head | Q8_0 | Shared with target |
The trunk retains its original imatrix-calibrated expert weights. The MTP block was quantized directly from Unsloth's Qwen3.6 BF16 export. Its down projection stays at Q4_K and the non-routed matrices stay at Q8_0. The published imatrix has no block-40 entries, so MTP IQ2_XXS quantization uses weight-energy importance rather than activation calibration.
| Type | Tensor count |
|---|---|
| F32 | 310 |
| Q8_0 | 320 |
| IQ2_XXS | 82 |
| Q2_K | 40 |
| Q4_K | 1 |
| Total | 753 |
MTP adds 332,777,472 bytes of weights. The embedding and output tensors are shared, keeping the added head below 350 MB.
Measured performance
AMD BC-250, RADV GFX1013, QuarkStar's optimized resident Vulkan path, greedy generation, Q8_0 K / Q4_0 V, 128 generated tokens, --mtp-draft 8 --mtp-margin 0:
| Context tokens | Ordinary decode | MTP decode | Speedup |
|---|---|---|---|
| 128 | 91.51 t/s | 97.01 t/s | 1.06× |
| 2,048 | 88.08 t/s | 131.39 t/s | 1.49× |
| 4,096 | 84.71 t/s | 105.27 t/s | 1.24× |
| 8,192 | 79.99 t/s | 97.08 t/s | 1.21× |
These are single serialized story sweeps. Loading, prefill and snapshot restoration are outside decode timing; GPU edge temperature peaked at 56°C. Results depend on hardware, prompts and runtime version.
The runtime verifies target rows together, retains decode routing arithmetic, uses confidence to limit recursive drafts, and can reuse an MTP-confirmed prompt suffix. The draft head uses a recent-target vocabulary mask; verification scores the full vocabulary.
Validation
- All 733 trunk tensor types, dimensions and payload hashes match the original model.
- Greedy MTP and sampled-boundary MTP replay passed at 128, 2K, 4K and 8K context with zero target-logit difference, including rejection rollback, eight-token acceptance and checkpoint restore.
- Short 16-thread CPU/Vulkan parity passed the existing top-1 through top-64 thresholds.
- Kernel tests cover batched MoE rows and masked Q8 head outputs.
At temperature zero, speculative tokens must match target argmax. At positive temperature, QuarkStar's default opportunistic mode samples block starts and accepts greedy tails, changing the sampling distribution. Use --no-mtp-opportunistic to retain ordinary per-token sampling.
File size: 12,070,095,168 bytes.
SHA-256:
4dab0f76b675cc1215b7ee599e85d2a5bbc97a20bd6fca18a8f35e343d90c013
Deprecated files
These earlier GGUFs remain available for existing installations and historical reference. They are superseded by Qwen3.6-35B-A3B-IQ2XXS-Q2K-Q8.gguf.
Their existing checksum and validation reports describe those deprecated releases.
| Deprecated file | Previous role |
|---|---|
Qwen3.6-35B-A3B-AntirezExperts-IQ2XXS-gateup-Q2K-down-Q8rest.gguf |
Original Q2 target without an integrated head |
Qwen3.6-35B-A3B-Layers34-39Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-Q8Rest-imatrix.gguf |
Last-six-layer Q4_K expert variant |
Qwen3.6-35B-A3B-MTP-Q4K-Q8_0-F32.gguf |
Separate MTP support GGUF |
The old q2-imatrix, q2-q4-imatrix, and mtp downloader targets have been removed. Use ./download_model.sh q2.
Acknowledgements
The asymmetric expert-quantization strategy and much of the runtime direction are inspired by Salvatore Sanfilippo's work on:
QuarkStar is a Vulkan-oriented redesign of ideas explored by DwarfStar, retargeted at Qwen3.6-35B-A3B and the AMD BC-250.
This work also depends heavily on the formats, quantization methods, kernels, tests and engineering knowledge developed by:
- llama.cpp
- GGML
- The Qwen team
- Salvatore Sanfilippo
- Georgi Gerganov
- llama.cpp and GGML contributors
License
The Qwen3.6-35B-A3B base model and these derived GGUF weights are distributed under the Apache License 2.0.
The q36 / QuarkStar inference-engine source code is distributed separately under the MIT License.
See the original model repository for the complete base-model license and usage terms: