An open model that tells you when it doesn't know.
Sentinel S1 Qwen3 8B is a LoRA adapter for Qwen/Qwen3-8B (revision b968826d9c46dd6066d109eabc6255188de91218), built by Sentovara. When the model is likely to be wrong it starts its reply with UNSURE:. A system around it can then look the answer up instead of trusting a guess. When it answers plainly, the answer is far more likely to be right than the original model's.
It uses the same recipe as Sentinel-S1-7B and Sentinel-S1-32B (Olmo), and shows that the recipe carries over to a model from a different lab.
Full method, results and limits: the write-up.
All checks were scored by script, on question sets kept out of training. The original model and Sentinel ran through the same code, with thinking mode off. The pass marks were written down before any results existed.
| Check | Original Qwen3 8B | Sentinel S1 Qwen3 8B | Pass mark | Result |
|---|---|---|---|---|
| Instruction-following (IFEval, 541 prompts) | 80.2% | 81.7% | within 2 points | Pass |
| Maths (GSM8K test, 1,319 problems) | 92.7% | 92.9% | within 2 points | Pass |
| Plain (unflagged) answers right (9,255 questions) | 24.1% | 54.6% | +10 points | Pass |
| Own wrong answers flagged | none | 83.2% | at least half | Pass |
| Replies end cleanly | 99.6% | 99.5% | at least 95% | Pass |
| Flagged questions right after searching (6,439) | 9.8% as a guess | 45.4% | search beats guess | Pass |
| Plain answers right in conversation (350 fresh chats) | 52.7% | 79.6% | +10 points | Pass |
| Own mistakes flagged in conversation | none | 86.7% | at least half | Pass |
| Ordinary chats flagged | — | 3 of 50 | at most 10% | Pass |
9 of 9 checks passed. The sign-off questions came from Natural Questions (validation), SimpleQA, GSM8K (test) and IFEval.
What changed and what didn't. Sentinel learned nothing new about the world. Across all test questions it gets about the same share right as the original. The gain is in judgement: it marks the answers it is likely to get wrong, so its plain answers can be trusted far more, and its flags show exactly when to look something up.
Version history, kept on record. The first Qwen version passed 8 of 9: it kept the flagging but lost 5.5 points of instruction-following. Its general practice conversations had come from another lab's data in a different style. In this version, every piece of training data was written by Qwen3 8B itself, and instruction-following came back above the original. Both results are in results/.
UNSURE:.The evaluation code is in eval/ and the result summaries behind the tables above are in results/.
Want the training recipe, or to work with us? The training code isn't published, but we're happy to talk. Open a discussion in the Community tab of this page or get in touch at sentovara.com.
Keep thinking mode off (enable_thinking=False): Sentinel was trained and tested that way.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base, rev = "Qwen/Qwen3-8B", "b968826d9c46dd6066d109eabc6255188de91218"
tok = AutoTokenizer.from_pretrained(base, revision=rev)
model = AutoModelForCausalLM.from_pretrained(base, revision=rev, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "Sentovara/Sentinel-S1-Qwen3-8B")
msgs = [{"role": "user", "content": "Who composed the music for the 1968 film 2001: A Space Odyssey?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=128, do_sample=False)
reply = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
if reply.startswith("UNSURE:"):
print("Sentinel is unsure. Look this up before trusting it:", reply)
else:
print(reply)
With vLLM, enable LoRA (enable_lora=True, max_lora_rank=16) and pass chat_template_kwargs={"enable_thinking": False}. Tested with transformers 5.22.1, peft 0.21.2 and vLLM 0.31.0.
Search step. For a flagged question, retrieve Wikipedia passages, number them, and ask Sentinel with this prompt, followed by the numbered passages and then Question: ...:
Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the number of the passage you used.
Our search took the top 50 BM25 passages from each of two Wikipedia indexes (the 2018 DPR index and a November 2023 dump), reranked them with BAAI/bge-reranker-v2-m3, and passed the top 10 to Sentinel. The code is in eval/.
eval/ifeval/), SimpleQA (MIT), and prompts from Ai2's Dolci instruction data (ODC-BY).Built by Sentovara. Designed, run and documented by Claude (Anthropic) for Sentovara.
Qwen/Qwen3-8B