openjev-fincalc
Try it in your browser → (side by side with openjev v5)
A 4B NLI judge that checks the arithmetic behind financial claims: whether a stated LTV, DSCR, interest cover, leverage, margin, capital ratio, net debt or covenant test actually follows from the evidence.
Conceived and built by Tony Esposito. Personal open-source project; not affiliated with any employer or client.
Why
LLM-as-a-judge cascades use small NLI models to check answers against evidence cheaply and escalate only the uncertain cases to a frontier model. In a banking judge, openjev handled lookups well but was confidently wrong on derived figures. On FinCalc-NLI the recommended openjev-4B v5 checkpoint scores 63% and catches only 30% of arithmetic slips; the original 4B checkpoint (the base of this fine-tune) labels almost every derived claim "neutral" (31%).
What is here
| File | Purpose |
|---|---|
gen_fincalc.py |
Generates FinCalc-NLI. Labels are computed in code, never annotated. |
train.py |
HF Jobs script: evaluate base, LoRA fine-tune with SNLI/MNLI/ANLI replay, merge, evaluate, push. |
Data (FinCalc-NLI)
- premise: evidence in four layouts (prose, bullets, table, key/value) with distractor companies, prior-year figures and policy lines.
- hypothesis: a derived figure, a covenant pass/breach, or a plain lookup.
- label: entailment (correct up to the claim's own rounding), contradiction (arithmetic slip, inverted ratio, another entity's figures, unit error, flipped covenant test, misquoted figure) or neutral (an input is missing, or the subject is not in the evidence).
- Test split: disjoint company, town and template pools, plus one formula (NPL coverage) never seen in training.
All data is synthetic. Companies, banks, properties and figures are invented. Results on FinCalc-NLI measure the targeted skill; they are not evidence of performance on real credit files. Validate on your own data before any production or model-risk use.
Run
python gen_fincalc.py --out data
hf upload <you>/fincalc-nli data . --repo-type dataset --private
hf jobs uv run --flavor a100-large --timeout 4h --secrets HF_TOKEN train.py -- --n-fincalc 30000 --n-replay 10000
Results
FinCalc-NLI test split (4,000 items: unseen names and templates), every openjev checkpoint evaluated with
eval/eval_fincalc.py on openjev's own NLIScorer:
| FinCalc-NLI | all | correct figure | arithmetic slip | inverted ratio | wrong entity | unit error | covenant | unseen formula |
|---|---|---|---|---|---|---|---|---|
| openjev-4B v5 (recommended) | 63.2% | 76% | 30% | 47% | 33% | 25% | 65% | 55% |
| openjev-4B v2 | 58.2% | 61% | 34% | 46% | 39% | 47% | 17% | 40% |
| openjev-0.8B v2s-long | 50.5% | 70% | 12% | 20% | 7% | 41% | 32% | 43% |
| openjev-4B original (base of this model) | 31.9% | 4% | 0% | 3% | 0% | 11% | 0% | 19% |
| openjev-fincalc-4b | 94.4% | 86% | 96% | 94% | 96% | 98% | 100% | 69% |
General NLI, 2,000 items, original base vs this model: MNLI matched 90.4% -> 89.6%, ANLI r3 41.8% -> 53.5%.
Training: LoRA r=32 on all text-backbone linear layers plus the score head, 30k FinCalc + 10k SNLI/MNLI/ANLI replay, 1 epoch, merged. About 40 minutes on one A100 through Hugging Face Jobs.
Limits (read before use)
- Synthetic data only, one language (English), a fixed set of ten metrics. Real credit files are messier.
- Transfer to an unseen formula is partial (about 70%), and three-operand metrics (net leverage) are weaker (73%).
- ANLI r3 train was part of the replay mix, so the ANLI gain is in-distribution, not a general reasoning gain.
- Use it as the cheap first stage of a judge cascade that escalates uncertain verdicts to a larger model, not as the only check on figures that feed a credit decision.
Usage
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
repo = "anespo28/openjev-fincalc-4b"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).eval()
premise = "Credit file extract\n- Harbour Foods Ltd: EBITDA £8.4m\n- Harbour Foods Ltd: net interest expense £2,100,000"
claim = "Harbour Foods Ltd has an interest cover ratio of 4.0x."
enc = tok(f"Premise: {premise}\nHypothesis: {claim}", return_tensors="pt")
p = torch.softmax(model(**enc).logits.float(), -1)[0] # [contradiction, entailment, neutral]
print(dict(zip(["contradiction", "entailment", "neutral"], p.tolist())))
The adapter alone is in lora/.
Credits and licence
Base model: AlexWortega/openjev (MIT), a Qwen3.5-4B NLI cross-encoder. This work is released under MIT.