Qwen3.5-27B-heretic GGUF

GGUF quantizations of llmfan46/Qwen3.5-27B-heretic.

Available quantizations

Multimodal projector (vision)

How to use

llama-server (recommended for API usage)

llama-server \
    -m Qwen3.5-27B-heretic-Q8_0.gguf \
    --mmproj Qwen3.5-27B-mmproj-BF16.gguf \
    --host 0.0.0.0 \
    --ctx-size 32768 \
    --gpu-layers 99 \
    --tensor-split 1,1 \
    --chat-template-kwargs '{"enable_thinking": false}' \
    --repeat-penalty 1.0 \
    --presence-penalty 1.5 \
    --min-p 0.0 \
    --top-k 20 \
    --top-p 0.8 \
    --temp 0.7

llama-cli

llama-cli -m Qwen3.5-27B-heretic-Q4_K_M.gguf -ngl 99 -c 4096

Which quantization should I use?

• Q8_0: Best quality, needs ~28 GB VRAM. Fits on 2x RTX 3090.
• Q6_K: Great quality, needs ~22 GB VRAM. Fits on 1x RTX 3090.
• Q5_K_M: Good balance of quality and size. Fits on 1x RTX 3090 with room for context.
• Q4_K_M: Most popular choice. Fits on 1x RTX 4070 Ti Super / RTX 3090.
• Q3_K_M: For constrained VRAM setups (~13 GB needed).

Quantization details

• Quantized with llama.cpp convert_hf_to_gguf.py + llama-quantize
• Original model: llmfan46/Qwen3.5-27B-heretic