An open model that tells you when it doesn't know.
Sentinel S1 32B is a LoRA adapter for allenai/Olmo-3.1-32B-Instruct (revision ac0587e4a7744a551c059d8cd17ba220bc940dae), built by Sentovara. When the model is likely to be wrong it starts its reply with UNSURE:. A system around it can then look the answer up instead of trusting a guess. When it answers plainly, the answer is far more likely to be right than the original model's.
Full method, results and limits: the write-up.
All checks were scored by script, on question sets kept out of training. The original model and Sentinel ran through the same code. The pass marks were written down before any results existed.
| Check | Original 32B | Sentinel S1 32B | Pass mark | Result |
|---|---|---|---|---|
| Instruction-following (IFEval, 541 prompts) | 85.6% | 85.0% | within 2 points | Pass |
| Maths (GSM8K test, 1,319 problems) | 94.7% | 94.7% | within 2 points | Pass |
| Plain (unflagged) answers right (9,255 questions) | 28.3% | 68.2% | +10 points | Pass |
| Own wrong answers flagged | none | 87.5% | at least half | Pass |
| Replies end cleanly | 99.6% | 99.6% | at least 95% | Pass |
| Flagged questions right after searching (6,448) | 10.9% as a guess | 45.2% | search beats guess | Pass |
| Plain answers right in conversation (350 fresh chats) | 55.0% | 77.9% | +10 points | Pass |
| Own mistakes flagged in conversation | none | 78.8% | at least half | Pass |
| Ordinary chats flagged | — | 3 of 50 | at most 10% | Pass |
9 of 9 checks passed. The sign-off questions came from Natural Questions (validation), SimpleQA, GSM8K (test) and IFEval.
What changed and what didn't. Sentinel learned nothing new about the world. Across all test questions it gets about the same share right as the original. The gain is in judgement: it marks the answers it is likely to get wrong, so its plain answers can be trusted far more, and its flags show exactly when to look something up.
Against the 7B Sentinel. The recipe was run again unchanged on the 32B, to test whether it carries over to a bigger model. It does. The 32B starts out knowing more: 32.5% of general-knowledge tries were right before training, against 14.9% for the 7B. So it answers plainly more often and flags a slightly smaller share of its mistakes (87.5% against 94.0%). It is the stronger of the two at instructions, maths and answering after a search. The 7B version is at Sentovara/Sentinel-S1-7B.
UNSURE:. Every answer it learned from on those questions was its own.eval/ and the result summaries behind the tables above are in results/.Want the training recipe, or to work with us? The training code isn't published, but we're happy to talk. Open a discussion in the Community tab of this page or get in touch at sentovara.com.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base, rev = "allenai/Olmo-3.1-32B-Instruct", "ac0587e4a7744a551c059d8cd17ba220bc940dae"
tok = AutoTokenizer.from_pretrained(base, revision=rev)
model = AutoModelForCausalLM.from_pretrained(base, revision=rev, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "Sentovara/Sentinel-S1-32B")
msgs = [{"role": "user", "content": "Who composed the music for the 1968 film 2001: A Space Odyssey?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=128, do_sample=False)
reply = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
if reply.startswith("UNSURE:"):
print("Sentinel is unsure. Look this up before trusting it:", reply)
else:
print(reply)
The adapter also works with vLLM's LoRA support (enable_lora=True, max_lora_rank=16). Tested with transformers 5.22.1, peft 0.21.2 and vLLM 0.31.0.
Search step. For a flagged question, retrieve Wikipedia passages, number them, and ask Sentinel with this prompt, followed by the numbered passages and then Question: ...:
Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the number of the passage you used.
Our search took the top 50 BM25 passages from each of two Wikipedia indexes (the 2018 DPR index and a November 2023 dump), reranked them with BAAI/bge-reranker-v2-m3, and passed the top 10 to Sentinel. The code is in eval/.
eval/ifeval/), SimpleQA (MIT), and Ai2's Dolci instruction data (ODC-BY).Built by Sentovara. Designed, run and documented by Claude (Anthropic) for Sentovara.
allenai/Olmo-3.1-32B-Instruct