AlexAtomic/qwen35-4b-GGUF

Base model: Qwen/Qwen3.5-4B

Qwen3.5 4B, self-quantized to GGUF by Atomic Chat. Built straight from Qwen's original weights with a per-tensor importance matrix. Runs fully offline.

Highlights

Efficient hybrid architecture combining Gated Delta Networks with sparse Mixture-of-Experts for high-throughput inference at low latency and cost.
Unified vision-language foundation trained with early fusion on multimodal tokens (these GGUF quants cover the text path).
262,144-token native context, extensible up to ~1,010,000 tokens.
Global linguistic coverage: Qwen reports support for 201 languages and dialects.
Thinking and instruct modes, each with its own recommended sampling presets.
Full quant ladder with an importance matrix on every quant over calibration_datav3.

[!NOTE] These GGUFs are self-quantized from the original weights, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model.

[!IMPORTANT] Always pass --jinja so the Qwen3.5 4B chat template is applied. Without it the model can emit malformed turns.

Model Overview

Property	Value
Base model	`Qwen/Qwen3.5-4B`
Total parameters	4B
Layers	32
Context length	262,144 native, extensible up to ~1,010,000
Architecture	Causal LM with vision encoder; hybrid Gated DeltaNet + Gated Attention, hidden dim 2560, trained with MTP
This repo	GGUF quants (imatrix), text path

Scores are Qwen's published results for the base Qwen/Qwen3.5-4B. Quantization preserves the large majority of this; Q4_K_M and up sit within a point or two of full precision.

Choosing a quant

Quant	Size	Notes
`Q2_K`	1.9 GB	Smallest. Minimal RAM, clear quality drop.
`IQ3_M`	2.2 GB	Beats Q3 at similar size thanks to imatrix. Best low-RAM pick.
`Q3_K_M`	2.3 GB	Low quality but usable.
`Q3_K_L`	2.4 GB	A step above Q3_K_M.
`IQ4_XS`	2.5 GB	Excellent quality for size. Recommended low-bit.
`Q4_K_S`	2.6 GB	Compact Q4, fast.
`Q4_K_M`	2.7 GB	Recommended default. Best balance of size, speed and quality.
`UD-Q4_K_XL`	2.9 GB	Dynamic. Embeddings and output kept at Q8_0 for higher quality at a Q4 footprint.
`Q5_K_S`	3.0 GB	Higher quality.
`Q5_K_M`	3.1 GB	Higher quality, low loss.
`Q6_K`	3.5 GB	Near lossless.
`Q8_0`	4.5 GB	Effectively lossless, reference quality.

[!TIP] Pick the largest file that fits your (V)RAM with room for context. Q4_K_M or UD-Q4_K_XL is the sweet spot for most setups; Q6_K or Q8_0 for maximum fidelity.

Get started

Run Qwen3.5 4B locally with:

Atomic Chat: the easiest path. Open the app, search AlexAtomic/qwen35-4b-GGUF, pick a quant, hit Use this model.
llama.cpp: llama-server -hf AlexAtomic/qwen35-4b-GGUF:Q4_K_M --jinja -c 8192
Ollama: ollama run hf.co/AlexAtomic/qwen35-4b-GGUF:Q4_K_M
LM Studio / Jan: search the repo id, download any quant.

Best practices

Parameter	Value
temperature	0.7
top_p	0.8
top_k	20
min_p	0.0
presence_penalty	1.5
repetition_penalty	1.0

Qwen's recommended Instruct (non-thinking) settings. Thinking mode for general tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.

Run in llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server

./llama.cpp/build/bin/llama-server \
    -hf AlexAtomic/qwen35-4b-GGUF:UD-Q4_K_XL \
    --jinja -ngl 99 -c 8192 -fa on

How these were made

Download Qwen/Qwen3.5-4B (original weights).
Convert to f16 GGUF with llama.cpp.
Build an importance matrix over calibration_datav3 (100 chunks).
Quantize the full ladder with --imatrix.
UD-Q4_K_XL additionally pins the token-embedding and output tensors to Q8_0.

License

Released by Qwen under the Apache 2.0 license. Quantized by Atomic Chat.