An open model that tells you when it doesn't know.
Sentinel S1 7B is a LoRA adapter for allenai/Olmo-3-7B-Instruct (revision 6e5971d9eba42665f5bd5a0fcf047f299ce1dccc), built by Sentovara. When the model is likely to be wrong it starts its reply with UNSURE:. A system around it can then look the answer up instead of trusting a guess. When it answers plainly, the answer is far more likely to be right than the original model's.
Full method, results and limits: the write-up.
All checks were scored by script, on question sets kept out of training. The original model and Sentinel ran through the same code. The pass marks were written down before any results existed.
| Check | Original 7B | Sentinel S1 7B | Pass mark | Result |
|---|---|---|---|---|
| Instruction-following (IFEval, 541 prompts) | 80.2% | 80.4% | within 2 points | Pass |
| Maths (GSM8K test, 1,319 problems) | 91.4% | 91.3% | within 2 points | Pass |
| Plain (unflagged) answers right (9,255 questions) | 20.3% | 75.1% | +10 points | Pass |
| Own wrong answers flagged | none | 94.0% | at least half | Pass |
| Replies end cleanly | 99.5% | 99.5% | at least 95% | Pass |
| Flagged questions right after searching (7,386) | 7.1% as a guess | 25.1% (basic search), 41.4% (stronger search) | search beats guess | Pass |
| Plain answers right in conversation (350 fresh chats) | 43.0% | 82.3% | +10 points | Pass |
| Own mistakes flagged in conversation | none | 92.5% | at least half | Pass |
| Ordinary chats flagged | — | 3 of 50 | at most 10% | Pass |
9 of 9 checks passed. The sign-off questions came from Natural Questions (validation), SimpleQA, GSM8K (test) and IFEval. The stronger search was added afterwards. It passed two of its three pre-set checks: it fell short of a 95% mark for naming a source, and it was adopted for the gain in right answers.
What changed and what didn't. Sentinel learned nothing new about the world. Across all test questions it gets about the same share right as the original. The gain is in judgement: it marks the answers it is likely to get wrong, so its plain answers can be trusted far more, and its flags show exactly when to look something up.
A larger version is available: Sentovara/Sentinel-S1-32B.
UNSURE:. Every answer it learned from on those questions was its own.eval/ and the result summaries behind the tables above are in results/.Want the training recipe, or to work with us? The training code isn't published, but we're happy to talk. Open a discussion in the Community tab of this page or get in touch at sentovara.com.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base, rev = "allenai/Olmo-3-7B-Instruct", "6e5971d9eba42665f5bd5a0fcf047f299ce1dccc"
tok = AutoTokenizer.from_pretrained(base, revision=rev)
model = AutoModelForCausalLM.from_pretrained(base, revision=rev, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "Sentovara/Sentinel-S1-7B")
msgs = [{"role": "user", "content": "Who composed the music for the 1968 film 2001: A Space Odyssey?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=128, do_sample=False)
reply = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
if reply.startswith("UNSURE:"):
print("Sentinel is unsure. Look this up before trusting it:", reply)
else:
print(reply)
The adapter also works with vLLM's LoRA support (enable_lora=True, max_lora_rank=16). Tested with transformers 5.22.1, peft 0.21.2 and vLLM 0.31.0.
Search step. For a flagged question, retrieve Wikipedia passages, number them, and ask Sentinel with this prompt, followed by the numbered passages and then Question: ...:
Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the number of the passage you used.
Our search took the top 50 BM25 passages from each of two Wikipedia indexes (the 2018 DPR index and a November 2023 dump), reranked them with BAAI/bge-reranker-v2-m3, and passed the top 10 to Sentinel. The code is in eval/.
eval/ifeval/), SimpleQA (MIT), and Ai2's Dolci instruction data (ODC-BY).Built by Sentovara. Designed, run and documented by Claude (Anthropic) for Sentovara.
allenai/Olmo-3-7B-Instruct