SearchJev-0.8B
SearchJev is a fast and calibrated System-1 model for search agents. Given a search state and a decision schema (e.g. is this document relevant?, is the evidence sufficient?, where should the next query go?), it returns a calibrated probability distribution over the legal values of every field in a single forward pass, without autoregressive decoding. In a dual-system search agent it takes the short decisions around a System-2 LLM and hands low-confidence ones back to it.
This repository holds the LoRA adapter of SearchJev-0.8B on Qwen/Qwen3.5-0.8B, together with its per-question-type temperatures and option-label alphabet.
- Code (training and inference): https://github.com/EvoScientist/Search_Jev
- Training data: SearchDecision-Bench
- Training logs: W&B report
- Paper: SearchJev: A Fast and Calibrated System-1 Model for Search Agents
Usage
git clone https://github.com/EvoScientist/Search_Jev && cd Search_Jev
pip install -r requirements.txt # a CUDA GPU is required (Qwen3.5 linear-attention kernels)
from huggingface_hub import snapshot_download
from training.inference import JEVPredictor
predictor = JEVPredictor(snapshot_download("SearchJev/SearchJev-0.8B"), device="cuda", merge_lora=True)
result = predictor.decide(
state={"question": "When was the Eiffel Tower completed?",
"documents": ["The Eiffel Tower was completed in March 1889 for the World's Fair."]},
question="Is the evidence sufficient to answer the question now?",
question_type="noul", # noul (yes/no) | choice | score
)
print(result.choice, result.confidence, result.probabilities)
The calibrated temperatures in calibration.json are applied automatically
(choice 1.11, yes/no 0.88, ordinal score 0.96).
Results
In-distribution test sets of SearchDecision-Bench (the decision-quality table of the paper). Relevance: NDCG@10 / ECE; other families: accuracy / ECE (all x100); latency: p50 ms per decision at batch size 1 on one H200. AR: the same backbone before training, emitting JSON under constrained decoding.
| Model | Relevance | Sufficiency | Routing | Navigation | Rewriting | Verification | Latency (ms) |
|---|---|---|---|---|---|---|---|
| Qwen3.5-0.8B AR (JSON) | 85.7 / 15.7 | 49.9 / 24.2 | 24.7 / 24.0 | 42.0 / 19.7 | 40.6 / 32.1 | 56.8 / 9.5 | 144.2 |
| SearchJev-0.8B | 98.6 / 5.8 | 89.4 / 5.1 | 83.2 / 5.0 | 56.8 / 4.6 | 84.0 / 2.3 | 91.7 / 9.3 | 27.6 |
Training
LoRA (rank 16, alpha 32, dropout 0.05) on the attention, Gated DeltaNet, and MLP projections; one epoch over the
683,130 training decisions of SearchDecision-Bench (5,176 steps, effective batch 132 on 6 H200 GPUs); AdamW,
learning rate 5e-5, cosine schedule with 5% warm-up, bf16; loss = soft-label cross-entropy + 0.5 x Brier score,
with half weight for LLM-assigned labels, demonstrations, and constructed rewrites; one temperature per question
type fitted on 5,000 validation decisions. config.resolved.yaml is the exact configuration of the run.
License
Apache-2.0, like the Qwen3.5 base model.