Sign up

ISTA-DASLab/Qwen3.5-4B-GGUF-GSQ

🤗 On Hugging Facetext-generationapache-2.01.9 GBGGUF✓ HF checksums availableupdated today
Magnet

Qwen3.5-4B — GGUF Q2_K_XL refined with GSQ

GGUF Q2_K_XL checkpoint of Qwen/Qwen3.5-4B

in which the discrete grid assignments have been refined with GSQ

(Gumbel-Softmax Quantization), starting from the public Unsloth GGUF

initialization and projected back into the same K-Quant format.

The optimized file runs unchanged on llama.cpp / Ollama and is a

drop-in replacement for the corresponding standard GGUF checkpoint.

Quantization details

  • Base model: Qwen/Qwen3.5-4B
  • Starting point: Unsloth GGUF UD-Q2_K_XL
  • Released quantization: Q2_K_XL
  • Format: GGUF K-Quant
  • Pipeline: Unsloth GGUF init → GSQ Gumbel-Softmax refinement → re-pack into the same format
  • Runtime: llama.cpp, ollama, LM Studio, anything that consumes GGUF

Storage layout

This file is bit-for-bit standard GGUF Q2_K. GSQ only changes the

values of the quantized weights by relearning the discrete grid assignments

inside each K-Quant block.

As a result:

  • The file size matches the corresponding upstream Q2_K_XL file for the same model.
  • Any llama.cpp / ollama build that loads a regular Qwen3.5-4B Q2_K_XL GGUF

should load this file with zero changes.

  • The Hugging Face UI reports the GGUF block type, e.g. Q2_K_XL, rather than

per-tensor dtypes. This refers to the on-disk K-Quant encoding, not the

precision of any optimizer state used during GSQ refinement.

Evaluation

We compare the GSQ-refined checkpoint against the Unsloth UD-Q2_K_XL

checkpoint used as initialization.

| Model | AIME 25 | GPQA Diamond | IFEval | GSM8K | MMLU-Pro

|---|---:|---:|---:|---:|---:|

| Unsloth UD-Q2_K_XL | 26.67 | 56.06 | 76.14 | 79.61 | 66.78

| GSQ-refined Q2_K_XL | 60.00 | 67.17 | 82.97 | 88.48 | 70.87

Evaluation notes

All evaluations are zeroshot. The Unsloth UD-Q2_K_XL model was evaluated with thinking disabled on all

tasks, because its thinking mode was found to be broken and produced worse

results.

For the GSQ-refined model:

  • AIME 25 and GPQA Diamond were evaluated with thinking enabled.
  • IFEval, GSM8K, and MMLU-Pro were evaluated with thinking disabled.

Token Efficiency

For MMLU-Pro, both models were evaluated with a maximum output budget of 32K

tokens.

  • The Unsloth model consumed 55.32M output tokens in total.
  • The GSQ-refined model consumed 41.82M output tokens in total.

Usage with llama.cpp

hf download ISTA-DASLab/Qwen3.5-4B-GGUF-GSQ \
  Qwen3.5-4B-Q2_K_XL.gguf --local-dir .

./llama-cli -m Qwen3.5-4B-Q2_K_XL.gguf -p "Hello"