Loading…
Loading…
Same BCE cross-encoder recipe as rectom-causal-reasoning-crossencoder-bce
(the {'MATCH@1': 0.064, 'Hit@1': 0.064, 'Hit@3': 0.218, 'Hit@5': 0.305, 'MRR': 0.179} 266-item model), retrained
on a 709-item leakage-safe Case Bank (docs129 large-scale
expansion) to test whether scaling causal-path supervision improves
reasoning-aware retrieval over frozen semantic (BGE) retrieval.
Group-disjoint 5-fold CV results (bare-dialogue_id grouping, fixing a grouping-key bug found in docs129): { "crossencoder_frozen": { "n": 709, "MATCH@1": 0.06770098730606489, "Hit@1": 0.06770098730606489, "Hit@3": 0.17489421720733428, "Hit@5": 0.2679830747531735, "MRR": 0.16755665214761337, "per_task_hit3": { "belief": 0.49019607843137253, "desire": 0.18110236220472442, "judgement": 0.10483870967741936, "prediction": 0.09375, "intention": 0.11764705882352941 }, "per_task_n": { "belief": 102, "desire": 127, "judgement": 248, "prediction": 96, "intention": 136 } }, "crossencoder_finetuned": { "n": 709, "MATCH@1": 0.08180535966149506, "Hit@1": 0.08180535966149506, "Hit@3": 0.2059238363892807, "Hit@5": 0.2947813822284908, "MRR": 0.1802352172741932, "per_task_hit3": { "belief": 0.3333333333333333, "desire": 0.14960629921259844, "judgement": 0.22580645161290322, "prediction": 0.07291666666666667, "intention": 0.22058823529411764 }, "per_task_n": { "belief": 102, "desire": 127, "judgement": 248, "prediction": 96, "intention": 136 } } }
McNemar (fine-tuned vs frozen): {'ft_only_wins': 100, 'frozen_only_wins': 78, 'n_discordant': 178, 'p_value': np.float64(0.11523195131755465)} Bootstrap 95% CI (Hit@3 delta): {'point_delta': 0.03102961918194641, 'ci_95_lo': -0.004231311706629048, 'ci_95_hi': 0.06770098730606489}
No causal path, gold answer, Direct/Causal outcome, or Benefit/Harm label is ever used as model input -- supervision-only, via the pair label.
cross-encoder/ms-marco-MiniLM-L6-v2