ymoslem/ModernBERT-base-TeleMath-router-qe-classifier-binary-10ep-lr2e-05-k1-instruct-5runs

Quality estimator for the Stage 2 cascade of CRE-Router on TeleMath, for Qwen3-4B-Instruct-2507. No-clustering ablation (k = 1) under the TPOT budget of 20 ms: gates every question Stage 1 sends to Qwen3-4B-Instruct; rejected answers escalate to Qwen3-30B-Thinking.

Given an efficient model's answer, it predicts whether to accept it (label 1) or escalate the query to a stronger model (label 0). An answer is escalated when the probability of acceptance is below 0.5.

Input format

question [SEP] full_output [SEP] num_tokens

full_output is truncated to its last 1,000 words before tokenisation, and the tokeniser runs at max_length=4096.

Training

ModernBERT-base, up to 10 epochs, lr 2e-5, batch size 64, max_length 4096, class-weighted cross-entropy. The training set is the model's own answers on the 299 TeleMath training questions, five sampled runs each (1,495 rows: 798 accept, 697 escalate), labelled by task correctness. Training stops early, with patience 4, and keeps the checkpoint with the best macro-F1 on one run of the 201 TeleMath test questions (87 accept, 114 escalate); the kept checkpoint is epoch 8 of 10, macro-F1 0.730.

Intended use

Deployed only where Stage 1 routes to Qwen3-4B-Instruct-2507. It is trained on that model's own outputs and does not transfer to another model's outputs.