Solar-Decision-JEV-27B-V2-NVFP4-FP8

NVFP4/FP8 quantization of SargeDev/Solar-Decision-JEV-27B-V2 (the V2 r32 + solar-1894 merge), built with llm-compressor 0.14.0. 52 GiB bf16 → 29 GiB, serving on GB10-class hardware in vLLM.

Recipe (Recipe B)

• NVFP4 W4A4 on attention q/k/v/o + MLP gate/up projections
• FP8_DYNAMIC on down_proj
• lm_head, embed_tokens, and the MTP block left in higher precision
• Calibration: 512 samples = 384 JEV decision prompts (the real workload format) + 128 UltraChat everyday rows, max seq 2048

Serve

vllm serve SargeDev/Solar-Decision-JEV-27B-V2-NVFP4-FP8 \
  --served-model-name solar-jev-27b-v2-nvfp4 \
  --quantization compressed-tensors \
  --trust-remote-code --max-model-len 8192 \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Pass enable_thinking: false or the template emits its reasoning trace into the answer.

What the quantization does to the numbers

The gate table (27,695 rows) lives on the bf16 card (SargeDev/Solar-Decision-JEV-27B-V2): choice top-1 0.828, score top-1 0.742, noul AUROC 0.983, overall 0.839 — below r32-alone (0.869), as documented there. This quant was calibrated on the real decision workload to keep that as tight as quantization allows, but it was not re-gated separately: read the bf16 numbers as the model's, and treat this build as "same behavior, ~half the footprint". If you need a typed decision instrument at maximum fidelity, use the bf16 repo.

Lineage & credits

• AutoTrust AI — r16/r32 decision LoRA recipe + 24-slot head + machinery (autotrust/JEV-27B).
• huihui-ai — abliterated Qwen3.8-27B backbone.
• SargeDev — training corpora: jev-distill-corpus-v3 (questions/states) and solar-decisions-corpus-v4 (the calibrated typed-decision labels the solar half trained on, labeled by upstage/solar-decide-flash).
• Qwen — base architecture.

Built on 2× NVIDIA GB10 (DGX Spark). Apache-2.0.