Solar-JEV-Gemma-embed-2

A calibration-probe model: the frozen text encoder from google/embeddinggemma-2 with three small linear heads trained to decode calibrated decision judgments — noul (yes/no with confidence), choice (best option with probability distribution), and score (0–5 rubric score with distribution) — from compact task/state descriptions.

The heads are distilled from upstage/solar-decide-flash (via the OpenRouter Decisions API), using the paired teacher-probability bank in SargeDev/solar-decisions-corpus-v4.

What this model is (and is not)

This is an experiment, not a production model. The body is fully frozen; only three linear heads were trained. The scientific question it answers:

Is Google's embedding space linearly decodable into calibrated, teacher-distributed judgments?

Answer: yes, substantially. 756k rows of Solar Decide Flash probability distributions were recovered at teacher-level agreement through ~120 KB of trained parameters (three Linear(768→{2,16,6}) heads).

It is not:

• a replacement for Solar Decide Flash
• a full fine-tune (see Limitations)
• wired into any production pipeline

Training

KL trajectory (train): 0.428 → 0.0285 — clean convergence, no late drift.

Results (inference-only, run once on each full split)

Overall

Per-kind (full test split)

Context for reading these numbers:

• Solar Decide Flash itself agrees with the v3 Jev-1.13 gold labels at ~58% argmax on choice — a different calibrated teacher is expected to disagree ~40–45% of the time. The probe's 55.8% overall agreement should be read against that backdrop, not against 100%.
• The v4 corpus spans 90 domains and 3 judgment kinds. This is a generalist probe; specialists (e.g. the cascade's production relevance student) reach 81.7% on their single narrow task.

Training dataset

Trained on SargeDev/solar-decisions-corpus-v4 (816,741 rows — v3 states re-labeled by Solar Decide Flash + 120k fresh-domain questions) and, via the v3 lineage, SargeDev/jev-distill-corpus-v3. Two independent teachers over overlapping state/question space (Jev 1.13 + Solar Decide Flash).

Intended use

• Research into whether decoder-style calibrated judgments are recoverable from frozen embedding spaces (what-if experiment).
• Baseline for future full-finetune runs (the 142h/3060 full-FT estimate for this corpus makes the frozen probe the right first experiment).
• Educational reference for KL-distillation of soft distributions onto frozen encoders.

Limitations

• Frozen body — head capacity is the ceiling. Choice/score tasks (8–16-way distributions) show the most headroom; noul (binary) is already near what a frozen-space linear head can express.
• Trained on one epoch; no hyperparameter sweep.
• MAXLEN 384 — long states are truncated.
• Choice vocabulary padded to 16 (the corpus-wide max) — real per-row option counts vary 2–16.
• Not calibrated against out-of-distribution inputs — the ood split was not evaluated (future work).

How to use

from huggingface_hub import snapshot_download
import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer, torch.nn

path = snapshot_download("SargeDev/Solar-JEV-Gemma-embed-2")
tok = AutoTokenizer.from_pretrained("google/embeddinggemma-2")
body = AutoModel.from_pretrained("google/embeddinggemma-2", dtype=torch.bfloat16).to("cuda")
heads = torch.load(f"{path}/heads.pt")
heads_n = torch.nn.Linear(768, 2).to("cuda"); heads_n.load_state_dict(heads["noul"])
heads_c = torch.nn.Linear(768, 16).to("cuda"); heads_c.load_state_dict(heads["choice"])
heads_s = torch.nn.Linear(768, 6).to("cuda"); heads_s.load_state_dict(heads["score"])

state = "A server room's cooling unit fails during peak load..."
question = "Given the server state, is immediate operator intervention required?"
enc = tok([f"task: classification | query: {question}\n{state}"],
          padding=True, truncation=True, max_length=384, return_tensors="pt")
with torch.no_grad():
    o = body(input_ids=enc["input_ids"].to("cuda"), attention_mask=enc["attention_mask"].to("cuda"))
    h = o.last_hidden_state
    m = enc["attention_mask"].to("cuda").unsqueeze(-1)
    h = (h * m).sum(1) / m.sum(1).clamp(min=1)
    noul_logits = heads_n(h.float())
    probs = F.softmax(noul_logits, dim=-1)  # [P(no), P(yes)]

Attribution

Corpus and method: SargeDev and contributors. The training bank comes from SargeDev/solar-decisions-corpus-v4 — 85% of the rows are the original SargeDev/jev-distill-corpus-v3 (Jev 1.13-labeled) questions/states re-labeled by Solar, plus 120k fresh Solar-labeled domains — making this model's training data a direct two-teacher lineage (Jev 1.13 states, Solar Decide Flash targets). If you use this model or its training corpus, please credit both prior datasets and link back.

Teacher: Solar Decide Flash (OpenRouter Decisions API, upstage/solar-decide-flash) labels the soft probability targets. Base encoder: google/embeddinggemma-2 (Apache-2.0 / Gemma Terms of Use — see base model license).

License

Apache-2.0. The frozen base encoder inherits its own upstream terms (Gemma); review those if redistributing.