A fine-tune of Laya that sorts requests to a voice assistant into the 150 intents of CLINC150, across banking, travel, cooking, work, small talk and more, and answers "none of these" to requests that are none of them. It gives a probability for every option in one forward pass, and its confidence can be trusted. So it can act on the requests it is sure about and send the rest to a fallback.
Requests a model was never built for are the hard part of any deployed assistant. CLINC150 was made to measure exactly that: its test holds 1,000 out-of-scope queries beside the 4,500 in-scope ones.
All 5,500 queries of CLINC150's "plus" test set, one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-18.
| Untuned Laya | This model | |
|---|---|---|
| In-scope queries right (of 4,500) | 2,352 (52.3%) | 4,277 (95.0%) |
| 95% confidence interval | 50.8 to 53.7% | 94.4 to 95.6% |
| Out-of-scope queries caught (of 1,000) | 490 (49.0%) | 716 (71.6%) |
| Mean stated confidence, all 5,500 | 0.666 | 0.955 |
| Calibration error (ECE, lower is better) | 0.251 | 0.050 |
It runs a little overconfident: it states 0.955 on average and is right on 90.8% of all 5,500 queries. Queries it scores 0.95 to 0.98 are right 93.1% of the time, and those at 0.98 or more 97.9%.
Acting only when sure. During the fine-tune, Laya held back 400 training queries and fitted a threshold on them, aiming at 99% right among the answers above it. That threshold is 0.9687. The model acts on a query only when it names one of the 150 intents at or above it; "none of these" and anything below it go to the fallback. Applied once to the test:
With the cut chosen on the test itself, which describes this test and is optimistic for new queries, and counting every answer above the cut, "none of these" included: 95% right while answering 88.1% of queries, 98% right at 36.4%, and 99% right only at 0.2% (13 queries).
Speed. One query alone: median 24 ms, 95th percentile 26 ms. In batches of 32: about 13 ms a query.
Against published results. The CLINC150 paper (Larson et al. 2019, Table 2, "OOS+" column, models trained with the out-of-scope queries as a class) reports BERT at 96.7% in-scope accuracy and 59.2% out-of-scope recall, and Rasa at 90.9% and 66.0%. Without the threshold, this model is 1.7 points behind BERT in scope (95.0%; BERT's number lies above this model's whole 95% interval) and 12.4 points ahead on out-of-scope recall (71.6%, interval 68.7 to 74.3%). It is ahead of Rasa on both. With the threshold it stops 91.7% of out-of-scope queries but acts on only 84.3% of in-scope ones, a trade the paper's models were not scored on. The paper does not say which size of BERT it used; Laya has 421 million parameters.
Where it is weakest. Out-of-scope queries: 284 of the 1,000 get an intent, most often directions (16), vaccines (13) and order (11). Among the intents, transactions and calendar are right 76.7% of the time each, then yes, restaurant reviews, how busy, pin change and report fraud at 80% each. The most common mix-up between two intents is transactions taken for spending history (6 of 30).
pip install laya==0.4.31
import json
import laya
from huggingface_hub import hf_hub_download
REPO = "VibeHash/laya-clinc150"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
questions = json.load(f)
result = model.predict("set a timer for ten minutes", questions, min_confidence=0.9687)
answer = result["answers"]["intent"]
if answer["abstention"] == "abstained" or answer["choice"] == "none of these":
print("fallback; best guess:", answer["choice"])
else:
print(answer["choice"], answer["answer_confidence"])
What it returned on the CPU:
| Query | Answer | Confidence | At 0.9687 |
|---|---|---|---|
| set a timer for ten minutes | timer | 0.9696 | acts |
| how much money is in my checking account | balance | 0.9776 | acts |
| book a table for two at 7 tonight | restaurant reservation | 0.9672 | fallback |
| what is the boiling point of mercury | none of these | 0.9724 | fallback |
| help me write a cover letter | none of these | 0.8896 | fallback |
The results above were measured on a GPU, and this table on the CPU. On the first 100 test queries the two gave the same answers; their confidences differed by 0.0002 at the median, but by 0.07 for one query. A query close to the threshold can fall on either side of it depending on the hardware.
questions.json exactly as written: the same instruction and the same 151 options in the same
order. labels.json maps each option back to CLINC150's label name; "none of these" is the label oos.min_confidence is the threshold fitted in training. Leave it out to get an answer for every query, or pick your
own cut from the trade-off above.predict_batch(texts, questions, batch_size=32) scores many queries in shared forward passes.expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"} to laya.load. SHA256SUMS lists every file.Laya's own laya.train.finetune at its default settings (4 epochs, effective batch 64, RLCD loss), with two
changes, both fixed before any fine-tuned model was scored:
head_max_len 512 instead of 192. The 151 options take 471 tokens; at 192 Laya cuts them short until many read
the same. The longest query in the data is 37 tokens, so no query is cut.target_error 0.01 instead of 0.10, so the threshold aims at 99% right.The options are CLINC150's label names with underscores written as spaces; oos is written "none of these". The
training set holds 15,250 queries (100 for each intent and 250 out of scope); Laya trained on 14,850 and held back
the other 400 to fit its temperature and the threshold. Training took 50 minutes on one RTX 4070 SUPER. The
validation split was not used. The test was read only by the scorer, through laya.load(...).predict_batch, the
same call a user makes. 2 of the 5,500 test queries also appear in the training data; they are left in, as in the
official split.
The code that built and scored this model is in code/. Data, weights and runs go under ~/.laya-tasks/.
hf download VibeHash/laya-clinc150 --include "code/*" --local-dir laya-clinc150
cd laya-clinc150/code
pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU
pip install -r requirements.txt
python -m layatasks.data clinc150 # CLINC150 at a pinned revision; writes and locks the test
python -m layatasks.train clinc150 --seed 0 # 50 minutes on an RTX 4070 SUPER
python -m layatasks.evaluate clinc150 --model ~/.laya-tasks/clinc150/runs/seed0/model --name seed0
Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed.
eval.json holds the full scores behind this card, train.json the training record.
Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint,
doi:10.20944/preprints202611.0781.v1. Trained on CLINC150
(CC BY 3.0): Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction,
EMNLP-IJCNLP 2019. See NOTICE.
convaiinnovations/laya