Solar-Decision-JEV-27B

The r32 + solar merge. huihui Qwen3.8-27B-abliterated + the AutoTrust r32 lm_head LoRA + our solar r16 LoRA, merged by direct LoRA-delta arithmetic (bit-exact verified: err = 0.000e+00 against a replicated computation, on sampled weights including lm_head).

The decision readout is trained INTO the model, so a typed decision is a single prefill with max_tokens=1, constrained to the option tokens, read back as log-probabilities. No separate head to run.

What this merge is, honestly

Both LoRAs are applied on top of the same base, and they are near-orthogonal (mean cosine 0.003) — so they do not cancel. But they are not additive in accuracy. On the same 27,695 rows, same harness, same head + calibration:

The combined row lands between its two parents on every metric, not above the better one. Stacking the solar adapter on top of a readout-trained model degraded the readout r32 had learned: choice top-1 fell 5.3 points and score KL roughly doubled.

We publish this anyway, and we publish those numbers, because the result is the finding. If you want the best decision instrument, use SargeDev/Solar-JEV-27B-R32 (the r32 merge on its own) or the already-published SargeDev/JEV-27B-Uncensored. If you want the merge of the decision readout with our own solar training, that is this repo.

On the chat side, this merge is also not the best

The solar r16 LoRA is what we trained on the JEV System-1 corpus (1,219 steps, KL+RPS decision-head distillation), and on its own it is a good everyday-chat model. Stacked on top of r32, it does not help:

The combined model is the weakest of the four on GSM8K and on instruction-following. It is the only row that loses on every axis.

So: this repo is a complete, honestly-measured merge of the two adapters, and on our measurements it is beaten by each of its parents. We publish it because the negative result is real and reproducible, and because the comparison belongs in the open. If you want the best model, use SargeDev/Solar-JEV-27B-R32 (or the already-published SargeDev/JEV-27B-Uncensored); if you want the best everyday-chat model, use SargeDev/Solar-JEV-27B (the solar build).

Quick start (vLLM, one GB10/Spark-class box)

vllm serve SargeDev/Solar-Decision-JEV-27B \
  --served-model-name jev-judgment-combined --trust-remote-code \
  --max-model-len 8192 --gpu-memory-utilization 0.85 \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Decision gate (AutoTrust-style, test_set_30k with yuri_v1 D1 placeholders excluded)

Read this table carefully, because it contains two very different comparisons.

• The sibling merge is our other build, scored on the same 27,695 rows with the same harness and the same head + calibration. It is a sibling model, not a quality bar — but it is the honest apples-to-apples reference for what the lm_head LoRA is worth.
• AutoTrust reference is the number published by the authors of the original AutoTrust work for their model on this task set. This is the bar.

On the typed-decision kinds (multiple-choice choice, ordinal score) this model is still below the published AutoTrust reference, and the gap is large - not a rounding difference. On the binary noul kind the gap is narrower. Cells marked "(below bar)" differ from the published reference by more than 5 points.

What this means: the fine-tune produces a usable decision instrument and a clearly better everyday-chat model than its base (see the benchmark matrix below), but it is not a reproduction of AutoTrust-level typed-decision accuracy. Treat the choice and score columns as an open result, not a solved one. We are publishing the numbers as measured, including the gap.

Harness validation. These numbers were produced by our own HTTP gate client, so we checked it against a known path: re-gating the r32 merge reproduced the reference's own numbers to within 0.001 on every metric (choice top-1 0.8704 -> 0.8693, score KL 0.0623 -> 0.0619, noul AUROC 0.9920 -> 0.9919). The harness is sound, so the gaps above are real.

Ablation: where the decision readout has to live

All four rows below are scored on the SAME 27,695 rows, with the SAME harness and the SAME decision head + calibration. The only difference is where the decision distribution is produced.

Two findings, in order of how much they matter:

1. The readout has to be trained INTO the model. A separate projection over the final hidden state (the solar r16 row) reaches only 0.5054 choice top-1; the same backbone with an lm_head LoRA in the training loop reaches 0.8693 on the identical rows. A separate projection can work, but it needs the backbone to cooperate, and a backbone trained without the readout in the loop does not.

2. Adding a second, readout-free adapter on top makes the typed-decision accuracy WORSE, not better. The combined row is the r32 merge plus the solar r16 delta applied on top (bit-exact arithmetic, verified). It lands between the two parents on every metric - 0.8162 choice top-1, between r32's 0.8693 and solar's 0.5054 - rather than above r32. The two LoRA deltas are near-orthogonal (mean cosine 0.003), so they do not cancel, but solar's delta still degrades the readout r32 trained.

We publish the combined row because it is a real, reproducible result, but the r32+lm_head merge is the better model and is the one this release recommends. We are not going to present a merge as an improvement when our own numbers say it is a regression.

Benchmark matrix (lm-eval-harness 0.4.13, greedy, same prompts for every row)

Questions scored per task:

Note: GSM8K scored with flexible-extract (the model answers in its own format). MMLU subset subjects: high_school_mathematics, college_biology, professional_psychology, jurisprudence, machine_learning, astronomy, global_facts, logical_fallacies (5-shot, _generative variants). MMLU --limit applies PER SUBJECT, so the subset is 8 x per-subject-limit questions.

Reproducibility: decoding is greedy (do_sample=False, temperature 0, seed 1234), but the server batches concurrent requests, and batch composition changes floating-point reduction order. Repeat runs of the same config can therefore differ by a point or two on the small-n tasks. The error bars in the table are the harness's own stderr over the scored rows.

Lineage & credits

This release is a descendant of the earlier SargeDev/JEV-27B-Uncensored and SargeDev/JEV-27B-VL-Uncensored repos. The credit chain is unchanged and is reproduced here in full:

• AutoTrust AI — Blocks-of-Experts recipe, the System-1 decision LoRA (r16/r32) + 24-slot decision head, and the serve_decide.py decision harness, from autotrust/JEV-27B and autotrust/JEV-27B-VL (Apache-2.0). The r32 adapter used here is theirs, unmodified: the backbone LoRA with the decision head re-expressed as an lm_head LoRA, so only the 24 verbalizer rows of the output projection change.
• huihui-ai — the abliterated (uncensored) Qwen3.8-27B backbone used here (Huihui-Qwen3.8-27B-abliterated, Apache-2.0; layers 18-51 ablated, vision tower untouched).
• SargeDev — jev-distill-corpus-v3, the 740,957-row calibrated typed-decision corpus AutoTrust's JEV models were trained on (Apache-2.0), and the solar r16 adapter trained here on solar-decisions-corpus-v4 (the 1,219-step JEV System-1 run, 17-domain tokenized mix).
• Qwen — Qwen3.8-27B base (Apache-2.0).
• TypeSafe AI — Jev 1.13, the original closed teacher behind the System-1 typed-decisions framing (referenced; not redistributed).

Base: huihui-ai/Qwen3.8-27B-abliterated (bf16; the text-only builds strip the vision tower and serve as a plain qwen3_5_text causal LM, the VL build keeps the tower and serves as qwen3_5 / Qwen3_5ForConditionalGeneration).

• Adapters — this release ships two, and they are NOT interchangeable:
  • solar r16 (alpha 32, 400 modules, no lm_head): trained 1,219 steps on the JEV System-1 corpus (solar-decisions-corpus-v4 tokenized mix: T1 gold domains + memory_relevance + 17-domain fill), KL+RPS decision-head distillation objective (AutoTrust-style), cosine LR 2e-4, KL_W 1.0, RPS_W 0.3. Merged with peft merge_and_unload; merge fidelity verified by re-gating the bare merged model (it reproduced the adapter-mounted numbers).
  • r32 (alpha 64, 401 modules, includes lm_head): the backbone LoRA plus the 24-slot decision head re-expressed as an lm_head LoRA (only the 24 verbalizer rows change), so a typed decision is a single prefill step with max_tokens=1 constrained to the option tokens.
• Decision head + calibration.json (shipped in this repo root): the instrument-panel piece agents call for numeric confidence — the 24-slot verbalizer id list, the per-slot bias, and the per-kind temperature. This is the SAME head and the SAME calibration file (verified byte-identical, md5 11359173… and 7f0f3914…) that produced every gate number in this card, so a consumer reproducing our numbers reproduces them exactly. For everyday natural-language judgment, chat with the model directly and set enable_thinking:false in chat_template_kwargs.
• Everything measured on one NVIDIA GB10 (DGX Spark-class); quantized with llmcompressor 0.14.0 (NVFP4) and llama.cpp (GGUF ladder).