ChainCheck-Judge-Qwen3.8-27B

An evidence-sufficiency judge for RAG whose score tracks whether the reasoning chain is intact, not merely whether the passages were edited (2Wiki chain selectivity +0.129 [+0.054, +0.200] → +0.461 [+0.397, +0.485] against the same-size zero-shot model).

Given a question and retrieved passages, the model returns a log-odds score for "the passages together are sufficient to answer". It is Qwen/Qwen3.8-27B with a LoRA trained on ChainCheck counterfactual triplets, merged back into the base as a single set of full bf16 weights (the base's MTP tensors are carried over unchanged, so MTP speculative decoding still works).

Why it matters: an ordinary sufficiency benchmark only asks whether a system separates untouched passages from broken ones, and a near-perfect score there can come from reacting to any edit. In a RAG pipeline that means an edited-but-valid context gets flagged and a subtly broken chain gets through. ChainCheck adds a matched control that separates the two.

Measurements (2026-10-05, conditions stated)

Same prompt, same scoring pipeline, one B200 GPU. Pairs a 27B model answers closed-book are excluded. Intervals are 95% pair-bootstrap (2,000 resamples). Σ = chain effect − |edit effect|; a lower bound above zero means the score reacts more to breaking the chain than to editing the passages.

Honest caveats:

• The tuned model tends to score the consistently edited context (B) above the untouched one (A): the edit effect is negative in 5 of 6 cells (range -0.449 to +0.019). Σ subtracts |EE|, so this is already penalised in the table, but the score is not edit-invariant.
• On real entities the tuned model is not better everywhere: chain selectivity is lower than zero-shot on MuSiQue confirmation (+0.180 → +0.107). The intervals overlap, so this is not a measured regression, but it is not a gain either.
• With synthetic replacement entities, chain selectivity is lower after tuning than for the same-size zero-shot model on MuSiQue test (+0.388 → +0.125), MuSiQue confirmation (+0.418 → +0.051), 2Wiki replication (+0.397 → +0.170). The gains are on real entities; do not assume they transfer to entities the model has never seen.
• 2Wiki was never used for training or model selection, and MuSiQue confirmation is a held-out slice built after the protocol was frozen. The release gate (2Wiki Σ lower bound > 0 on real and synthetic, MuSiQue test real Σ lower bound > 0, 2Wiki nominal no more than 0.02 below zero-shot) was fixed before any score was seen.
• The zero-shot column is the paper's reference judge re-scored through this pipeline. It ranks items essentially identically to the paper's run, but A and B differ by one entity and score almost the same, so small bf16 numerical differences flip some comparisons and Σ moves within the paper's reported interval.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ThakiCloud/ChainCheck-Judge-Qwen3.8-27B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

SYSTEM = ("You are a strict evidence auditor for a retrieval-augmented QA system. "
          "Decide whether the given passages, taken together, contain enough information "
          "to fully answer the question. Related-but-insufficient passages do NOT count.")
USER = ("Question:\n{query}\n\nPassages:\n{passages}\n\n"
        "Do the passages together contain sufficient evidence to answer the question? "
        "Answer with a single word: yes or no.")

yes = [i for i in range(len(tok)) if tok.decode([i]).strip().lower() in {"yes", "y"}]
no = [i for i in range(len(tok)) if tok.decode([i]).strip().lower() in {"no", "n"}]

def sufficiency_score(query, passages):
    body = "\n\n".join(f"[{i + 1}] {p[:4000]}" for i, p in enumerate(passages))
    msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER.format(query=query, passages=body)}]
    text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
    with torch.no_grad():
        lp = torch.log_softmax(model(**ids).logits[0, -1].float(), -1)
    return (torch.logsumexp(lp[yes], -1) - torch.logsumexp(lp[no], -1)).item()  # > 0 leans "sufficient"

The tokenizer, vocabulary and architecture are identical to the base (standard load). The score is a log-odds, not a calibrated probability: choose the threshold on your own validation data and use it to gate generation (answer, retrieve more, or abstain).

Limitations

• English Wikipedia multi-hop questions only (MuSiQue, 2WikiMultiHopQA). The edits are entity substitutions on two-hop chains; numerical errors, temporal staleness and longer chains are not measured.
• A positive Σ says the score reacts more to chain breaks than to edits on this benchmark. It does not certify that answers built on accepted contexts are correct.
• Not a chat model. It is meant to be read through the yes/no logits shown above, not through free generation.

Benchmark

All three sizes, each next to its own zero-shot baseline:

• Qwen3.5-4B: released
• Qwen3.5-9B: released
• Qwen3.8-27B: released

Benchmark kit and standalone evaluator: ThakiCloud/ChainCheck.

Training

LoRA (rank 8, α 16, attention q/k/v/o on full-attention layers, lr 1e-4, gradient accumulation 8, seed 0, one epoch over 7,481 MuSiQue train triplets) with labels A = 1, B = 1, D = 0 and loss = binary cross-entropy + max(0, 1 − (min(s_A, s_B) − s_D)). After merging, the full weights were reloaded from disk and re-scored on 64 held-out items to confirm they reproduce the adapter scores.

License

Apache-2.0 (base: Qwen/Qwen3.8-27B). Training and evaluation passages derive from MuSiQue (CC BY 4.0) and 2WikiMultiHopQA (Apache-2.0); passage text originates from Wikipedia.

Paper

ChainCheck: When Near-Perfect Evidence-Sufficiency Scores Fail to Distinguish Chain-Sensitive Systems — preprint forthcoming

The paper shows that near-perfect nominal evidence scores do not reliably distinguish chain-sensitive systems, and that separating chain sensitivity from edit sensitivity changes the apparent ordering of systems. This model applies that protocol as a training signal. Read its gains against the same-size zero-shot column, not against the nominal score alone, which is close to its ceiling for every system.