Sign up

ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ

🤗 On Hugging Faceimage-text-to-textapache-2.035.5B params140 GBsafetensors✓ HF checksums availableupdated today
Magnet

Qwen3.6-35B-A3B — 2-bit GSQ

Low-bit quantization of Qwen/Qwen3.6-35B-A3B

(MoE, 35B total / 3B active) produced with GSQ

(Gumbel-Softmax Quantization).

The routed-expert MLP weights are quantized to 2-bit GSQ with an effective

storage cost of ≈2.13 bpp, while the attention layers, shared experts, and

LM head are quantized to INT8. The checkpoint preserves most of the base

model's reasoning, coding, and long-context behaviour while substantially

reducing its memory footprint.

Evaluation Results

We evaluate the quantized checkpoint against the original

Qwen/Qwen3.6-35B-A3B

across reasoning, instruction-following, science QA, and math benchmarks.

| Benchmark | Base Model | 2-bit GSQ |

|---|---:|---:|

| AIME 2025 | 93.33 | 93.33 |

| GPQA Diamond | 83.84 | 80.30 |

| IFEval | 91.25 | 87.77 |

| MMLU-Pro | 84.87 | 81.00 |

| GSM8K | 96.21 | 93.93 |

Quantization details

  • Base model: Qwen/Qwen3.6-35B-A3B
  • Routed-expert MLP precision: 2-bit GSQ, ≈2.13 bpp effective storage
  • Codebook: 2-bit symmetric scalar {-2, -1, 0, +1} × scale
  • Group size: 128
  • Additional INT8 quantization: self_attn, linear_attn, shared experts, and lm_head
  • Format: Humming
  • Pipeline: GPTQ initialization → Gumbel-Softmax refinement (Lion optimizer)
  • 2-bit quantized: routed-expert MLPs (gate_proj, up_proj, down_proj)
  • INT8 quantized: attention (self_attn, linear_attn), shared experts, and LM head
  • Kept in BF16: embeddings, layernorms, MoE routing gate, and other non-quantized components

Storage layout (why the HF UI shows I32 + I8 + BF16)

The Hugging Face "Tensor types" widget reports the container dtype of each

tensor stored in the safetensors checkpoint, rather than the effective

precision of the original model weights.

The routed-expert MLPs use the Humming exact-width 2-bit layout. For every

2-bit expert-MLP Linear with original weight shape

[out_features, in_features], the following tensors are stored:

| Tensor | Dtype | Shape on disk | Meaning |

|---|---|---|---|

| .weight | I32 | [out_features, in_features × 2 / 32] = [out_features, in_features / 16] | 2-bit values bit-packed along the input dimension, LSB-first: 16 weights per INT32 word |

| .weight_scale | BF16 | [out_features, in_features / 128] | One symmetric scale per group of group_size = 128 weights along the input dimension |

| self_attn / linear_attn / shared-expert / lm_head weights | I8 | implementation-dependent | INT8-quantized weights |

| Embeddings / norms / MoE routing gate / other unquantized tensors | BF16 | unchanged | Copied or retained at BF16 precision |

For the 2-bit GSQ expert weights, the effective storage is:

2 bits (packed) + 16 bits / 128 (group scale) ≈ 2.13 bpp

The checkpoint therefore uses a mixed-precision quantization layout:

2-bit GSQ for routed-expert MLP weights, INT8 for attention, shared experts,

and the LM head, and BF16 for the remaining unquantized components.

The quantization_config in config.json describes the Humming quantization

layout used by the checkpoint.

Note: GSQ training first writes shards in compressed-tensors
pack-quantized format, where the 2-bit codebook is padded into a 4-bit
INT32 container. The published checkpoint has been re-packed via
convert_to_humming.py into exact-width 2-bit Humming storage, hence the
2 / 32 shape factor for the 2-bit expert weights.

Serving with vLLM

Serving this checkpoint requires:

  • vLLM 0.27.1
  • the vLLM compatibility patch included in this model repository
  • an Ampere (SM ≥ 80) or Hopper GPU

Install the required vLLM version:

pip install vllm==0.27.1

Then, from the directory containing the model files, run the included

patch_vllm.py script:

python patch_vllm.py

The patch must be executed in the **same Python environment in which vLLM

0.27.1 is installed**. It patches the installed vLLM package with the changes

required to load and serve this checkpoint.

After applying the patch, the model can be served normally:

vllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ --reasonin-parser qwen3
Important: running the model with an unpatched vLLM installation is not
supported. If vLLM is reinstalled or the environment is recreated, run
python patch_vllm.py again before serving the model.
Model size / text-only usage: the full checkpoint is approximately
12.6 GB, including the MTP and vision components. These components are
optional for ordinary text-only generation: the vision weights are only
required for multimodal inputs, while the MTP weights are only required when
using MTP/speculative decoding. Removing both the vision and MTP weights
reduces the model size to approximately 9.9 GB.
Actual VRAM usage during serving will be higher than the raw model size and
depends on KV-cache allocation, context length, batch size, and vLLM runtime
overhead.

Citation

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026},
  url    = {https://arxiv.org/abs/2604.18556}
}