Solar-Decision-JEV-27B-VL-GGUF
GGUF ladder for the vision build of Solar Decision JEV 27B (the bf16 repo
SargeDev/Solar-Decision-JEV-27B-VL).
Sizes
| quant | size | decode tok/s (prompt/tokens) |
|---|---|---|
| IQ4_XS | 14.35 GiB | 952.57 / 13.96 |
| Q2_K | 10.11 GiB | 741.87 / 18.44 |
| Q3_K_L | 13.55 GiB | 788.35 / 14.06 |
| Q3_K_M | 12.56 GiB | 816.24 / 15.06 |
| Q3_K_S | 11.40 GiB | 799.62 / 16.27 |
| Q4_K_M | 15.65 GiB | 862.12 / 12.41 |
| Q4_K_S | 14.73 GiB | 894.15 / 13.32 |
| Q5_K_M | 18.18 GiB | 827.26 / 11.08 |
| Q5_K_S | 17.66 GiB | 838.98 / 11.50 |
| Q6_K | 20.88 GiB | 736.04 / 9.65 |
| Q8_0 | 27.04 GiB | 804.35 / 8.11 |
You need TWO files for a VL model
llama.cpp splits a vision model into the text/decoder weights and a separate multimodal projector:
Solar-Decision-JEV-27B-VL.Q4_K_M.gguf <- the text side
Solar-Decision-JEV-27B-VL.mmproj-Q8_0.gguf <- the vision tower -> LLM
Both are in this repo. mmproj-f16 is also included if you want the projector
at full precision.
Quick start
llama-server \
-m Solar-Decision-JEV-27B-VL.Q4_K_M.gguf \
--mmproj Solar-Decision-JEV-27B-VL.mmproj-Q8_0.gguf \
-ngl 99 --jinja
The honest headline
The decision-gate numbers for the combined merge are on the bf16 card. This ladder is the same weights in GGUF form. The combined merge's measured decision-gate accuracy is below its r32 parent (choice top-1 0.8162 vs 0.8693) - see the bf16 card for the full ablation. We publish the ladder because it is the artifact people actually run, with the numbers attached.