Qwen3.5 · 0.8B

EXL3  ·  3.0 bpw  ·  1.0 GB  ·  Dense


[!NOTE] An ExLlamaV3 build of Qwen/Qwen3.5-0.8B at 3.0 bits per weight. See Quants for sibling repos at other bit‑widths or browse the collection.

Quants

BPW     Mode     Size     median KL
self-sampled  		   Status
3.0 plain 1.0 GB 1.64e-04 link
3.0 SC H6 V6 1.0 GB 4.35e-05 this repo
4.0 plain 1.1 GB — link

Inference

Loader Use it for
TabbyAPI OpenAI‑compatible HTTP server. Drop‑in for OpenAI clients.
text‑generation‑webui Local chat UI. Pick the ExLlamaV3 loader from the model dropdown.
ExLlamaV3 Direct Python API for embedding the model in your own code or pipeline.

Download

pip install -U huggingface_hub

hf download \
  Honkware/Qwen3.5-0.8B-exl3-SC-3.0bpw-H6-V6 \
  --local-dir ./Qwen3.5-0.8B-exl3-SC-3.0bpw-H6-V6
Quantization recipe  (advanced, embedded in quantization_config.json)
Setting Value
Format EXL3
Bits per weight 3.0
Head bits 6
Calibration rows 250
Calibration data exllamav3 bundled mix (c4, code, multilingual, technical, tiny, wiki)
Codebook mul1
Out‑scales always
Parallel mode enabled
Vision tower 6 bits

License & use

[!IMPORTANT] Use and license follow the base model. Quantization adds no additional restrictions. Refer to the upstream repository for terms, citation, and safety documentation.

Quantized with BlockQuant  ·  convention {org}/{model}-exl3-{bpw}bpw