Qwen3.5-4B-M8 — GGUF

GGUF quantizations of ManniX-ITA/Qwen3.5-4B-M8.

See the BF16 model card for the recipe and full ablation matrix.

Quants

Usage

llama-server -m m8-Q6_K.gguf \
    --port 8099 -c 32768 -ngl 99 --no-warmup \
    --reasoning-format deepseek --reasoning-budget 8192 \
    --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0

Eval (Q6_K, lm-evaluation-harness)