EXL3 quantizations of ornith-ai/Ornith-1.5-9B for ExLlamaV3 / TabbyAPI. See the base model card for model details, prompt format, and sampling settings (coding temperature=0.6, top_p=0.95; general temperature=1.0, presence_penalty=1.5; tool parser Qwen3 XML).
| Branch | Decoder | lm_head |
Size | VRAM (text-only load) | Notes |
|---|---|---|---|---|---|
main / 3.0bpw |
3.0 | 6 | 6.5 G | 4.8 GiB | Default. Sized for 12 GB cards (RTX 3080 utility tier). |
2.5bpw |
2.5 | 6 | 6.0 G | 4.5 GiB | Deeper cut; marginal gain — the 16-bit embed/vision tail dominates. |
VRAM measured text-only with an 8K cache on an RTX 3080; on-disk size is larger because the vision tower ships 16-bit and is not loaded for text serving.
Same prompts and harnesses (promptfoo CRUXEval-O, 50 samples seed 42; QuixBugs repair/identify) run against this 3.0bpw quant (ExLlamaV3, RTX 3080 12 GB, Q8 KV) and an NVFP4 (ModelOpt 4-bit) build of the same base model (vLLM, fp8 KV, DGX Spark GB10). Single runs; judge-scored metrics carry judge variance; small-n — treat ±2 CRUXEval tests as noise.
| Eval | NVFP4 4-bit | EXL3 3.0bpw (this repo) |
|---|---|---|
| CRUXEval-O pass@1 | 86.0% (43/50) | 80.0% (40/50) |
| QuixBugs repair | 9/23 | 5/23 |
| QuixBugs identify (LLM-judged) | 14/23 | 9/23 |
| Tool-call classification | 5/6 | 5/6 |
The 2.5bpw branch is unevaluated — expect it at or below the 3.0bpw row.
ExLlamaV3 convert.py, 250x2048 calibration, mul1 codebook, lm_head 6bpw,
vision 16-bit. MTP: not included (base config declares one hidden MTP layer
but the base checkpoint ships no mtp.* tensors).
MIT, same as the base.
ornith-ai/Ornith-1.5-9B