Ornith-1.5-35B — Q8_0 mmproj (multimodal projector)

A Q8_0-quantized multimodal projector (mmproj) for ornith-ai/Ornith-1.5-35B-A3B-GGUF, requantized from the official BF16 projector.

Same vision tower. Same KV metadata. ~32% smaller — which is the whole point: it fits vision into VRAM budgets where the BF16 projector does not.

Saving: 275.3 MiB (32.0%).

Why this exists

On a 24 GB card serving Ornith-1.5-35B-Q4_K_M at a long context with --n-cpu-moe 0, the mmproj and its clip compute buffer compete for the last ~1 GiB of VRAM. The BF16 projector (861 MiB) does not fit; the Q8_0 projector (586 MiB) does, with headroom left for the clip warmup buffer. Concretely, for one measured profile (ctx 120000 · --parallel 4 · --n-cpu-moe 0):

free after model load                       ~1061 MiB
BF16 mmproj  861    + clip warmup 248.10  = 1109 MiB  -> does not fit
Q8_0 mmproj  586    + clip warmup 248.10  =  834 MiB  -> fits (~225 MiB spare)

When the projector does not fit, a vision request can abort the process rather than fail the request — so the quantized projector is not only a VRAM win, it is the difference between a working vision server and a crashing one.

How it was converted

Requantized BF16 → Q8_0 with mmproj-quant.py included in this repo:

PYTHONPATH=<llama.cpp>/gguf-py python3 mmproj-quant.py \
    mmproj-Ornith-1.5-35B-BF16.gguf \
    mmproj-Ornith-1.5-35B-Q8_0.gguf Q8_0

Conversion is purely local — no GPU, no back-conversion through a model loader.

Rules applied (ggml blocks run along ne0, the contiguous axis)

• BF16, ndim >= 2, ne0 % 32 == 0 → Q8_0 (83 tensors) — the attention/FFN weight matrices of the vision tower.
• Everything else → byte-identical passthrough (251 tensors):
  • 1-D norms and biases (F32),
  • v.position_embd.weight (F32, interpolated at runtime),
  • v.blk.*.ffn_down.weight — ne0 = 4304, and 4304 % 32 = 16, so it is not block-quantizable and is kept at full precision.

Why not llama-quantize?

llama-quantize loads through llama_model_load_from_file(), which does not know the clip architecture — the mmproj is read by the separate clip/mtmd loader — and refuses with "unsupported model architecture: 'clip'". An mmproj is, however, a plain GGUF, so this script requantizes it directly and preserves every KV field.

Verification

mmproj-verify.py compares this file against the BF16 source tensor-by-tensor (dequantize each Q8_0 block and compare against the original):

=== KV metadata ===
  src keys=26 dst keys=26 missing=[] extra=[]
  differing values: none

=== tensors ===
  src=334 dst=334  name-set equal: True
  passthrough identical: 251 ok / 0 bad
  Q8_0 dequant sanity  : 83 ok / 0 bad
  worst Q8_0 abs errors:
    max_abs=0.0022 mean_rel=0.00968  v.blk.0.attn_qkv.weight
    max_abs=0.0018 mean_rel=0.00600  v.blk.1.attn_out.weight
    max_abs=0.0018 mean_rel=0.00682  v.blk.1.attn_qkv.weight
    max_abs=0.0016 mean_rel=0.00584  v.blk.6.ffn_up.weight
    max_abs=0.0016 mean_rel=0.00582  v.blk.26.attn_out.weight

• KV metadata identical (26/26 keys, no differing values).
• 334/334 tensors present, name-set identical.
• 251 passthrough tensors byte-identical.
• 83 Q8_0 tensors: worst-case absolute error 0.0022, mean relative error < 1%.

Reproduce with:

PYTHONPATH=<llama.cpp>/gguf-py python3 mmproj-verify.py \
    mmproj-Ornith-1.5-35B-BF16.gguf mmproj-Ornith-1.5-35B-Q8_0.gguf

Usage

Drop-in replacement for the BF16 projector — only the --mmproj path changes:

llama-server \
    -m Ornith-1.5-35B-Q4_K_M.gguf \
    --mmproj mmproj-Ornith-1.5-35B-Q8_0.gguf \
    --host 0.0.0.0 --port 8080 \
    --ctx-size 120000 --n-gpu-layers 999 --parallel 4 ...

or with llama-mtmd-cli:

llama-mtmd-cli -m Ornith-1.5-35B-Q4_K_M.gguf \
    --mmproj mmproj-Ornith-1.5-35B-Q8_0.gguf \
    --image photo.jpg -p "Describe this image."

Built and tested against a recent llama.cpp master (b11379-era, Qwen-VL / clip arch). Any build that runs the BF16 projector runs this file unchanged.

Files

Provenance

• Source: mmproj-Ornith-1.5-35B-BF16.gguf from ornith-ai/Ornith-1.5-35B-A3B-GGUF (902,822,240 bytes).
• Output SHA256: c6150bccaae5673fe5f9d9318c9ebc7adda0c6fc74a4fd79214734ef2a873bb6

License

MIT, following the base model. See LICENSE.