An experimental Laya decision model that grades an existing selected citation occurrence. It answers three fixed questions:
Supply an existing candidate occurrence. The model returns recognition,
completeness and source-type decisions. Use the included questions.json
unchanged alongside the weights.
Use laya==0.4.2, transformers==4.57.1, PyTorch and safetensors. The
evaluated Spark runtime used NVIDIA PyTorch/CUDA; native training/evaluation used
FP16 autocast, whereas the public SDK used eager CUDA BF16. Predictions can vary
across devices and precisions. A separate CPU release smoke check tests package
loading and inference.
Load this package with the Laya SDK, using its native encoder/tokenizer and decision-head layout. The included helper provides source formatting, checksum verification and complete-input checks.
import json
from huggingface_hub import snapshot_download
import laya
# Pin the full immutable release commit from the repository's Files and versions.
folder = snapshot_download("thurstonpowers/xll-citation-verifier", revision="<release-commit>")
# inference.py is included in that folder; add folder to sys.path or copy/import it.
import sys
sys.path.insert(0, folder)
from inference import build_state, verify_package, predict
questions = verify_package(folder)
agent = laya.load(folder, device="cpu", backend="eager", compile=False, fast=False)
raw = "The applicant shall comply with section 4-1 of the Connecticut General Statutes."
quote = "section 4-1 of the Connecticut General Statutes"
start = raw.index(quote)
state = build_state(raw, start, start + len(quote), section="example")
response = predict(agent, questions, state)
print(json.dumps(response["answers"], indent=2))
The same helper supports device="cuda"; ensure CUDA exists rather than silently
accepting SDK fallback. The CLI rejects missing/invalid immutable Hub revisions
and verifies all release file hashes. Example for a downloaded local release:
python inference.py --model ./downloaded-model --input source.txt \
--start 32 --end 79 --section example --device cpu
Choose the actual occurrence's offsets; the CLI numbers above correspond to the example string only. Offsets use Python Unicode code-point indexing. Decode original UTF-8 bytes while preserving newlines. Keep the original text/quote unchanged and select the specific occurrence being graded.
The trained state includes containing jurisdiction/type and section identity,
the selected quote and original offsets, 48 immediate boundary characters on
each side, and an explicitly partial source passage (220 characters before,
260 after) with selected-span markers. Federal inputs use jurisdiction="United States" and the original containing-track expression. Current evaluation covers
Connecticut and federal material; other-state evaluation remains open.
The saved native budgets are 8,192 complete sequence tokens / 2,048 head tokens, with a native 48-token option-text cap. The helper checks every question's complete head/options, native fit and SDK truncation statistics. Overflow must be held explicitly. The verifier reads the selected span and bounded surrounding evidence passage.
Recognition/span labels are a, b, u (yes, no, uncertain); source-type labels
are defined in questions.json. Incomplete references may be recognition-positive
but span-negative. Numeric cuts, omitted governing qualifiers and range endpoints
are incomplete; complete coordinated list members can be usable.
The SDK returns a probability vector plus two different confidence measures:
answer_confidence is calibrated top-answer confidence; confidence is
entropy-derived. Use the calibrated answer confidence for confidence diagnostics
and thresholds. Applications must preserve
uncertainties, upstream extraction failures and known review holds independently.
Dataset: thurstonpowers/xll-citation-verifier, with the frozen optimization, calibration and evaluation splits and annotation provenance.
convaiinnovations/laya, revision
7b928d828b7b0e022f929d9bd2e44165aa270148, ModernBERT-large encoder.CT was excluded from the upstream generative extractor's training, but is included in this verifier's training. The verifier's original unseen CT reserve is identified separately below.
| Question | Agree / items | Agreement |
|---|---|---|
| Recognition | 2,565 / 2,572 | 99.73% |
| Usable span | 2,540 / 2,572 | 98.76% |
| Source type | 2,549 / 2,572 | 99.11% |
| All question items | 7,654 / 7,716 | 99.20% |
All three labels agreed for 2,525/2,572 cases. Span mistakes included 18 of 304 reference-negative spans accepted and 14 of 2,268 positives rejected. These scores measure agreement with provisional supervision. Expert-gold legal accuracy remains an open evaluation task.
The original 3,644-section reserve has zero document/chapter-group/exact-text overlap with optimization, calibration, internal evaluation and teacher-selected sources. All 8,929 prepared exact occurrences received valid SDK responses, with no missing outputs or truncation. Failed/empty source extractions remain in the original section denominator; 1,759 sections had zero candidates, 16 had failed or incomplete extraction, and 141 quote/input holds were preserved.
A uniform 100-candidate-occurrence sample, seed 2026110901, was individually assistant-reviewed and frozen before model answers were inspected:
| Question | Agree / scorable judgments |
|---|---|
| Recognition | 100 / 100 |
| Usable span | 98 / 100 |
| Source type | 98 / 99 |
One source-type taxonomy judgment was unresolved and excluded before scoring; the original sample remains 100. There were two false span approvals and no false span rejections. The type sample was mostly state regulations (43), state statutes (37), internal references (7) and Acts (6), with only two CFR references and one manual. Broader source-type coverage remains an open evaluation task.
Known mistakes: approving Public Law 90-321 while omitting explicit Title I
(confidence 0.9956); approving United States Code inside the publication title
United States Code Service; inferring state_statute for bare Chapter 60
without evidence establishing its source class (confidence 0.9929).
These are assistant-reference agreements from an occurrence-uniform sample. Some cases share source documents. The remaining 8,829 predictions are unreferenced. Whole-document citation recall, expert legal accuracy and nationwide generalization remain open evaluation tasks.
Treat this as experimental legal-data tooling. Rare source types, other-state generalization, ambiguous publication titles and evidence-sufficiency abstention need further evaluation. High confidence can accompany real errors. Official publications and qualified human legal review remain authoritative.
The release includes FP16 weights, tokenizer/encoder configuration, exact questions, inference contract/helper, aggregate evaluation, provenance and SHA-256 checksums.
Weights SHA-256:
efe5338fe2ec926ad6bf214dba0d03e3b5e584ec7767eacf3f44a93380e8847b.
Questions SHA-256:
bda0ecb8cda7f7502dce129de2fc8d6e92c3a28d924f15c6902b2483649db552.
Apache 2.0; derived from Convai Innovations' Laya and Answer.AI's ModernBERT.
See LICENSE, NOTICE and provenance/BASE_MODEL_CARD.md. Fine-tuning and release
documentation by Thurston Powers.
Training and evaluation data: thurstonpowers/xll-citation-verifier. Use the model-specific contract and splits described in the dataset card.
convaiinnovations/laya