Sign up

nerkyor/Qwen3.5-4B-GGUF-imatrix

🤗 Hugging Face sourcetext-generationapache-2.02.8 GBGGUF✓ HF checksums availableupdated today
No torrent yet →

Qwen3.5-4B Q4_K_M (imatrix) - Lynn calibration

This repository ships an imatrix-calibrated Q4_K_M GGUF of the official Qwen/Qwen3.5-4B BF16 weights, built on DGX Spark (GB10, sm_121) by the Lynn team.

This is a pure quantization of the upstream model. It is not a distillation and not a Lynn-native NVFP4/W4A8 checkpoint.

Files

File Size SHA256 Role
Qwen3.5-4B-Q4_K_M-imatrix.gguf 2.6 GB 7abaf02bbe25c608deb308db526766f761ad4fb85c512a69ff36520c4b304b23 llama.cpp / Ollama / LM Studio GGUF weights
Qwen3.5-4B.imatrix 3.5 MB 863a93c58a14925b58303a369d9bb155411b40d52fde121f195eaf7a6691f07c imatrix calibration data used for quantization

Build Details

  • Source weights: Qwen/Qwen3.5-4B official BF16
  • Converter: llama.cpp convert_hf_to_gguf.py --outtype f16
  • Calibration: Lynn Chinese + English + code mix, 256 chunks, 512 ctx
  • Quantizer: llama-quantize --imatrix Qwen3.5-4B.imatrix ... Q4_K_M
  • Built on: DGX Spark (GB10, sm_121), 2026-05-24

Evaluation

V8/V9, MMLU500, and GPQA Diamond thinking-on evaluations are running on Spark. Those artifacts and scores will be added in a follow-up update.

Run

llama-server \
  -m Qwen3.5-4B-Q4_K_M-imatrix.gguf \
  --host 0.0.0.0 --port 18099 \
  --ctx-size 32768 --n-gpu-layers 999 \
  --jinja --reasoning on

The GGUF embeds the upstream Qwen3.5 chat template, including thinking-mode support via chat_template_kwargs.enable_thinking.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B.