Thaki-CiteRank-4B (Research Preview)

Status: Research Preview. Validated for evidence-passage ranking, not for fact verification.

• What is shown: better ranking of evidence passages than the base Qwen3-Reranker-4B, including on a contract domain (CUAD) that was never used for training: nDCG@10 0.905 vs 0.838.
• What is not shown: that the model judges support, partial support or contradiction correctly in general. The 4-way head (SUPPORTS / PARTIAL / CONTRADICTS / IRRELEVANT) was evaluated only on synthetic technical-docs data built with the same generator as its training data. Do not present its output as a fact-check.
• Unsupported citations remain. When the model cites (top-1 above the dev threshold), the cited passage is not labeled as supporting evidence in 33.1% of CUAD groups, 12.6% of QASPER groups and 1.5% of docs groups. Do not use it to cite automatically without a human or another verifier in the loop.
• It can still cite a contradicting passage. In the bundled example, a passage that states the opposite value ("The default value is Never") scores -1.562, just above the dev threshold of -1.6875, so it would be cited even though the head labels it CONTRADICTS (0.90). Use the head only as a review flag and keep a human in the loop.
• No effect on live AI search engines has been measured.

At a glance

What it does

Given a question and/or a claim and a set of candidate passages, the model scores each passage by how well it can be cited as evidence. It keeps the official Qwen3-Reranker prompt format with a fixed instruction (see example_inference.py); use the same instruction, or scores will shift.

relevance is meant for ordering passages and for abstention with a threshold. The threshold used in the evaluation, τ = −1.6875, was chosen on the dev split by F1; re-tune it on your own data.

Bundled example output (example_inference.py, one B200, bf16):

+3.500 cite=yes head=SUPPORTS(1.00, uncalibrated)     | ... The default value is Always.
-1.562 cite=yes head=CONTRADICTS(0.90, uncalibrated)  | ... The default value is Never.   <- failure: cited
-5.688 cite=no  head=IRRELEVANT(0.99, uncalibrated)   | Services expose a set of Pods ...

The 4-way head is an auxiliary output. Its softmax values are not calibrated probabilities: we applied no temperature scaling, and the 10-bin expected calibration error of the top class was 0.048 on QASPER test and 0.046 on CUAD test (on those sets only SUPPORTS / IRRELEVANT labels exist). Treat them as scores.

Results

Test sets were scored once, after the pass/fail gates had been committed on the basis of dev-set baselines. All five pre-registered gates passed (QASPER +0.03 nDCG with CI above zero; CUAD CI above zero; docs Macro-F1 ≥ 0.70 and contradiction recall ≥ 0.85; docs contradiction-citation and abstention no worse than FactGuard-General-4B-v2 + 0.02; unsupported citation rate no higher than the base model in every domain). Confidence intervals are cluster bootstrap (2,000 resamples, clustered by paper / document family / contract).

Docs 4-way head (34,360 passages): Macro-F1 0.976 (FactGuard-General-4B-v2: 0.631), F1 SUPPORTS 0.972, PARTIAL 0.983, CONTRADICTS 0.952, IRRELEVANT 0.996, contradiction recall 0.990. Read the limitations below before using these numbers.

QASPER by answer type (nDCG@10): extractive 0.941, free-form 0.904, yes/no 0.854 (base: 0.903 / 0.838 / 0.743).

Limitations

1. The docs 4-way results are in-distribution and synthetic. Docs claims and labels were produced by an in-house Qwen3.8-27B model from Kubernetes and Python documentation, with the same generator for training and test (split by document family). PARTIAL claims were built by joining two claims with ", and", so the claim text itself reveals the group type. PARTIAL F1 0.983 does not measure real partial-support detection.
2. CUAD shows ranking transfer, not contradiction-detection transfer. CUAD has only relevant / not-relevant labels. No held-out human-labeled data was used to test the CONTRADICTS or PARTIAL outputs.
3. QASPER is lexically easy. Extractive answers share words with the evidence (BM25 R@5 0.954 on the extractive subset), and its evidence annotations are noisy (two-annotator evidence Jaccard about 0.49). Unannotated paragraphs are treated as irrelevant.
4. Unsupported citations remain frequent on CUAD (33.1%). Automatic citation without verification is not safe.
5. No live AI search effect is shown. Nothing here measures citation or exposure in ChatGPT, Gemini, Perplexity, Google AI Overviews or any other product.
6. English only. Passages longer than 1,024 tokens (prompt included) are truncated.

Training

One run on one NVIDIA B200 (3.1 hours), no hyperparameter search. LoRA r 16, alpha 32, dropout 0.05, lr 1e-4, 2 epochs (the first epoch was selected on dev), gradient accumulation 8, max length 1,024. Loss: pointwise BCE on the yes/no score (positive weight 4) + listwise cross-entropy over each candidate group + 0.5 × class-weighted cross-entropy for the 4-way head. Training data: QASPER train (2,069 questions) and docs train (10,144 groups), with BM25 hard negatives; at most 10 candidates per group. CUAD was not used for training or model selection. Full settings and per-split numbers: eval_manifest.json.

Data sources and licenses

RAGBench, ALCE and CC BY-SA sources were not used. No source text, question, claim or label is included in this repository. The documentation was split into sections, and claims, value-swapped contradictions and joined partial claims were generated from those sections as training and evaluation signal only. None of the dataset or documentation authors endorses this model.

This repository contains model weights, configuration, tokenizer files, cls_head.safetensors, this card, eval_manifest.json, example_inference.py and the Apache-2.0 license text. The license field covers this repository's contents and follows the base model; it is not a legal determination about the upstream texts.

@inproceedings{dasigi2021qasper,
  title={A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers},
  author={Dasigi, Pradeep and Lo, Kyle and Beltagy, Iz and Cohan, Arman and Smith, Noah A. and Gardner, Matt},
  booktitle={NAACL-HLT},
  year={2021}
}
@article{hendrycks2021cuad,
  title={{CUAD}: An Expert-Annotated {NLP} Dataset for Legal Contract Review},
  author={Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer},
  journal={NeurIPS Datasets and Benchmarks},
  year={2021}
}

Usage

from example_inference import load, score   # file in this repository
model, tok, head = load("ThakiCloud/Thaki-CiteRank-4B")
relevance, head_scores = score(model, tok, head,
    question="What is the default restart policy of a Pod?",
    claim="The default restartPolicy of a Pod is Always.",
    docs=["...passage 1...", "...passage 2..."])

Release verification

Before publication, every file was downloaded from the private Hub revision 59dcb630 in a separate process with an empty cache (all 11 files matched the uploaded package by sha256) and checked offline on one NVIDIA B200 against the original training checkpoint and the frozen scores behind the pre-registered verdict. The verdict itself was not recomputed and no new evaluation was run. Every check passed: