Solar-Decision-JEV-27B-VL-GGUF

GGUF ladder for the vision build of Solar Decision JEV 27B (the bf16 repo SargeDev/Solar-Decision-JEV-27B-VL).

Sizes

quant size decode tok/s (prompt/tokens)
IQ4_XS 14.35 GiB 952.57 / 13.96
Q2_K 10.11 GiB 741.87 / 18.44
Q3_K_L 13.55 GiB 788.35 / 14.06
Q3_K_M 12.56 GiB 816.24 / 15.06
Q3_K_S 11.40 GiB 799.62 / 16.27
Q4_K_M 15.65 GiB 862.12 / 12.41
Q4_K_S 14.73 GiB 894.15 / 13.32
Q5_K_M 18.18 GiB 827.26 / 11.08
Q5_K_S 17.66 GiB 838.98 / 11.50
Q6_K 20.88 GiB 736.04 / 9.65
Q8_0 27.04 GiB 804.35 / 8.11

You need TWO files for a VL model

llama.cpp splits a vision model into the text/decoder weights and a separate multimodal projector:

Solar-Decision-JEV-27B-VL.Q4_K_M.gguf     <- the text side
Solar-Decision-JEV-27B-VL.mmproj-Q8_0.gguf <- the vision tower -> LLM

Both are in this repo. mmproj-f16 is also included if you want the projector at full precision.

Quick start

llama-server \
  -m Solar-Decision-JEV-27B-VL.Q4_K_M.gguf \
  --mmproj Solar-Decision-JEV-27B-VL.mmproj-Q8_0.gguf \
  -ngl 99 --jinja

The honest headline

The decision-gate numbers for the combined merge are on the bf16 card. This ladder is the same weights in GGUF form. The combined merge's measured decision-gate accuracy is below its r32 parent (choice top-1 0.8162 vs 0.8693) - see the bf16 card for the full ablation. We publish the ladder because it is the artifact people actually run, with the numbers attached.