gemma · 4 · 31B · it · scotoma · 2
EXL3 · 5.0 bpw · 23.7 GB · Dense
[!NOTE] An ExLlamaV3 build of
ReadyArt/gemma-4-31B-it-scotoma-2at 5.0 bits per weight. See Quants for sibling repos at other bit‑widths or browse the collection.
Quants
| BPW | Head bits | Calibration rows | Size | Status |
|---|---|---|---|---|
| 4.0 | 8 | 250 | 20.1 GB | link |
| 4.5 | 8 | 250 | 21.9 GB | link |
| 5.0 | 8 | 250 | 23.7 GB | this repo |
Inference
| Loader | Use it for |
|---|---|
| TabbyAPI | OpenAI‑compatible HTTP server. Drop‑in for OpenAI clients. |
| text‑generation‑webui | Local chat UI. Pick the ExLlamaV3 loader from the model dropdown. |
| ExLlamaV3 | Direct Python API for embedding the model in your own code or pipeline. |
Download
pip install -U huggingface_hub
hf download \
Honkware/gemma-4-31B-it-scotoma-2-exl3-5.0bpw \
--local-dir ./gemma-4-31B-it-scotoma-2-exl3-5.0bpw
quantization_config.json)| Setting | Value |
|---|---|
| Format | EXL3 |
| Bits per weight | 5.0 |
| Head bits | 8 |
| Calibration rows | 250 |
| Codebook | mul1 |
| Out‑scales | always |
| Parallel mode | enabled |
The codebook is recorded in the weights, so loaders pick it up with no configuration. mul1 needs ExLlamaV3 v0.0.3 or newer; an older build ignores the marker and decodes the weights with the wrong codebook.
Loaded automatically by every ExLlamaV3 loader; reproduced here for searchability.
License & use
[!IMPORTANT] Use and license follow the base model. Quantization adds no additional restrictions. Refer to the upstream repository for terms, citation, and safety documentation.
Quantized with BlockQuant · convention
{org}/{model}-exl3-{bpw}bpw