Qwen3.5-35B-A3B-heretic GGUF

GGUF quantizations of brayniac/Qwen3.5-35B-A3B-heretic.

This is a Mixture of Experts (MoE) model with 35B total parameters and ~3B active parameters per token.

Available quantizations

Multimodal projector (vision)

How to use

llama-server (recommended for API usage)

llama-server \
    -m Qwen3.5-35B-A3B-heretic-Q8_0.gguf \
    --mmproj Qwen3.5-35B-A3B-mmproj-F16.gguf \
    --host 0.0.0.0 \
    --ctx-size 32768 \
    --gpu-layers 99 \
    --tensor-split 1,1 \
    --chat-template-kwargs '{"enable_thinking": false}' \
    --repeat-penalty 1.0 \
    --presence-penalty 1.5 \
    --min-p 0.0 \
    --top-k 20 \
    --top-p 0.8 \
    --temp 0.7

llama-cli

llama-cli -m Qwen3.5-35B-A3B-heretic-Q4_K_M.gguf -ngl 99 -c 4096

Which quantization should I use?

• Q8_0: Best quality, needs ~35 GB VRAM. Fits on 2x RTX 3090.
• Q6_K: Great quality, needs ~27 GB VRAM. Fits on 2x RTX 3090 with room for context.
• Q5_K_M: Good balance. Fits on 1x RTX 3090 (24 GB).
• Q4_K_M: Most popular choice. Fits on 1x RTX 3090 with context room.
• Q3_K_M: For constrained VRAM setups (~16 GB needed).

Quantization details

• Quantized with llama.cpp convert_hf_to_gguf.py + llama-quantize
• Original model: brayniac/Qwen3.5-35B-A3B-heretic