A Korean + English decision model. Give it a state (text or JSON: a conversation, a document, a tool resultโฆ) and typed questions โ yes/no (noul), pick-one (choice), or ordinal (score) โ and it returns a calibrated probability for every option in one forward pass, without generating text. Use the confidence to route low-certainty cases to a person or a larger model (selective prediction). From the same pass it reads two answers per question โ a trained decision head and the backbone's own answer letter โ and averages them.
ํ๊ตญ์ดยท์์ด ๊ฒฐ์ ๋ชจ๋ธ์ ๋๋ค. ์ํ(๋ํยท๋ฌธ์ยท๋๊ตฌ ๊ฒฐ๊ณผ ๋ฑ)์ ํ์ ์ด ์ ํด์ง ์ง๋ฌธ(์/์๋์ค, ํ๋ ๊ณ ๋ฅด๊ธฐ, ์์ ์ฒ๋)์ ๋ฐ์, ๊ธ์ ์์ฑํ์ง ์๊ณ ํ ๋ฒ์ forward๋ก ์ ํ์ง๋ง๋ค ๋ณด์ ๋ ํ๋ฅ ์ ๋ ๋๋ค. ๋ผ์ฐํ , ๊ฐ๋๋ ์ผ(๊ทธ๋๋ก ๋ณด๋ด๋ ๋๋), RAG ๊ด๋ จ์ฑ ํ๋จ, ์์ด์ ํธ ๋ถ๊ธฐ, ๋ฐ์ดํฐ ๋ผ๋ฒจ๋ง์ ์๋๋ค. ๊ฐ์ forward์์ ์ง๋ฌธ๋ง๋ค ํ์ตํ ๊ฒฐ์ ํค๋์ ๋ต๊ณผ ๋ฐฑ๋ณธ์ด ์ง์ ๊ณ ๋ฅธ ๊ธ์ ๋ต์ ํจ๊ป ์ฝ์ด ํ๊ท ํฉ๋๋ค.
enable_cuda_graphs() (63 ms without; mean of 5 requests) โ on par with laya (English) at 30 ms, 56% faster than clef-flash (9B) (73 ms), 72% faster than Prompted Qwen3.5-4B (115 ms) and 77% faster than JEV-9B (141 ms). Longer states and more questions add little (25 โ 44 ms from simple to complex requests); models that send one request per question grow with each (Prompted Qwen3.5-4B 94 โ 146 ms, JEV-9B 86 โ 225 ms). Only laya-multilingual is faster (23 ms), at 30โ36% accuracy on the eval set.state and questions as decide() to POST https://api.blackmallangcow.com/v1/decide (docs).| Model | Size | Held-out ko | Held-out en | Eval set ko | Eval set en | ms / request |
|---|---|---|---|---|---|---|
| nari-decision-4b (this model) | 4B | 85.9 | 85.3 | 82.1 | 76.9 | 32 |
| Prompted Qwen3.5-4B | 4B | 68.8 | 62.6 | 67.8 | 64.7 | 115 |
| clef-flash (9B) | 9B | 75.2 | 69.8 | 73.8 | 71.8 | 73 |
| JEV-9B | 9B | 67.3 | 53.2 | 76.0 | 67.3 | 141 |
| laya-multilingual | 322M | 51.0 | 49.9 | 36.4 | 30.1 | 23 |
| laya (English) | 395M | โ | 52.1 | โ | 36.5 | 30 |
Accuracy (%) after temperature calibration, same questions for every model (on held-out, Prompted Qwen3.5-4B and JEV-9B skip questions with more options than they can take). ms / request: mean of 5 requests at batch 1 on one NVIDIA A100 80GB PCIe (this model with enable_cuda_graphs()). Details under Evaluation.
์์ฝ
enable_cuda_graphs()๋ฅผ ์ผ ๊ฐ์ด๊ณ ๋๋ฉด 63ms, 5๊ฑด ํ๊ท ). laya (English) 30ms์ ๋น์ทํ๊ณ , clef-flash (9B) 73ms๋ณด๋ค 56%, Prompted Qwen3.5-4B 115ms๋ณด๋ค 72%, JEV-9B 141ms๋ณด๋ค 77% ๋น ๋ฆ
๋๋ค. ์ง๋ฌธ์ด ๋๊ณ ์ํ๊ฐ ๊ธธ์ด์ ธ๋ 25ms โ 44ms๋ก ์กฐ๊ธ๋ง ๋์ด๋ฉ๋๋ค. ์ด๋ณด๋ค ๋น ๋ฅธ ๊ฑด 23ms์ธ laya-multilingual๋ฟ์ด๊ณ , ํ๊ฐ์
์ ํ๋๋ 30โ36%์
๋๋ค.decide()์ ๊ฐ์ ์ํ์ ์ง๋ฌธ์ POST https://api.blackmallangcow.com/v1/decide๋ก ๋ณด๋ด๋ฉด ๋ฉ๋๋ค (๋ฌธ์).from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("blackmallangcow/nari-decision-4b", trust_remote_code=True)
model = AutoModel.from_pretrained("blackmallangcow/nari-decision-4b", trust_remote_code=True).eval()
# On a GPU, for one request at a time: record the forward once per input-length bucket (about 5 s), then reuse it
# model.to("cuda").enable_cuda_graphs()
out = model.decide(
tok,
state="๊ณ ๊ฐ: ์ง๋์ฃผ ์ฃผ๋ฌธ์ด ๋ ๋ฒ ๊ฒฐ์ ๋์ด์. ํ ๊ฑด์ ์ทจ์ํด ์ฃผ์ธ์.",
questions={
"refund": {"type": "noul", "instructions": "ํ๋ถยท์ทจ์ ์์ฒญ์ธ๊ฐ์?"},
"team": {"type": "choice", "instructions": "๋ด๋น ํ์ ๊ณ ๋ฅด์ธ์.",
"options": ["billing", "shipping", "account", "none"]},
"urgency": {"type": "score", "instructions": "๊ธด๊ธ๋", "options": ["1", "2", "3", "4", "5"]},
},
)
# {"answers": {"refund": {"answer": True, "p_true": ..., "probs": {...}},
# "team": {"answer": "billing", "confidence": ..., "probs": {...}},
# "urgency": {"answer": "3", "confidence": ..., "expected": ..., "probs": {...}}},
# "truncated": False}
instructions is optional; the question name and options alone also work. Korean or English.truncated tells you). Questions and options are never cut.paths: the decision head's and the backbone letter's probabilities before averaging and calibration. Questions with more than 26 options use the head only.transformers with Qwen3.5 support. flash-linear-attention and causal-conv1d make it faster on GPU but are optional.Give each question a name that says what it asks. The name is written into the input, and without instructions it is the whole question:
### ์ง๋ฌธ 1: q1 [์/์๋์ค (true/false)] <- the model cannot tell what is asked
### ์ง๋ฌธ 1: contains_pii [์/์๋์ค (true/false)] <- the name alone asks the question
q1 โ contains_pii (does the text expose personal information?)flag โ safe_to_send (can this reply go to the user as is?)cat โ support_team (which team should handle it?)level โ urgency (how urgent, 1โ5?)Add a none option when none of the options may apply.
Keep options mutually exclusive.
Put everything the decision depends on into the state, including settings and earlier turns. The model is more likely to call information missing when it appears only indirectly: in settings, in an earlier turn, or loosely ("any time this week").
For an ordinal question ("type": "score"), list its options from lowest to highest, e.g. ["1", "2", "3", "4", "5"] or ["low", "medium", "high"]. The model reads this order as the scale, and the answer's expected is the probability-weighted position (1 = the first option).
Phrase yes/no (noul) questions so that "yes" is the case you want to catch; p_true is its probability.
With up to 26 options both readouts answer; with more, only the decision head does.
Use the probability, not only the top answer: act automatically above a threshold and send the rest to a person or a larger model.
Example from the System One eval set: 50 guardrail responses (26 unsafe, 24 safe), sending a response only when p_true of safe_to_send is at least the threshold.
| Threshold | Unsafe responses sent | Safe responses held for a person |
|---|---|---|
| 0.5 | 2 / 26 | 1 / 24 |
| 0.8 | 1 / 26 | 1 / 24 |
| 0.9 | 1 / 26 | 4 / 24 |
The sample is small. Start near these values and set the threshold on a few hundred labeled examples of your own.
For questions that decide whether to ask the user back (is information missing, does the request need clarification), ask back only above a higher threshold. Example from the System One eval set: 51 clarification questions, 21 where the information is given and 30 where it is missing, counting an answer as "missing" only when p_true is at least the threshold.
| Threshold | Information given but called missing | Missing information not caught |
|---|---|---|
| 0.5 | 8 / 21 | 2 / 30 |
| 0.7 | 5 / 21 | 3 / 30 |
| 0.8 | 4 / 21 | 4 / 30 |
Again a small sample: tune the threshold on your own requests.
Let the model choose the next tool, but enforce prerequisite steps such as authentication or looking up an ID in your agent code. The model tends to pick the tool for the user's goal rather than the step that has to come first: on the API-Bank tool-selection task of the Decision Index public suite, when the correct next call was the authentication step (GetUserToken), it chose it in 2 of 108 cases (256 of 508 correct overall).
For ordinal (score) questions, read expected or the whole distribution rather than only the top option: on held-out the top option is exactly right 57.7% of the time, but within one step 91.9%.
Without revision, from_pretrained follows main, which moves with each release. In production, pin one: revision="v3.2".
Probabilities are scaled by temperatures fitted on the calibration split, kept in model.config.temperatures. On data unlike the training mix, re-fit them: set model.config.temperatures = {"*": 1.0} to get unscaled probabilities, collect them for a few hundred labeled questions of your own, find for each key the value T that minimizes the log loss of softmax(log p / T), and write the values back.
Keys are type:options (noul:2, choice:3-5, choice:6+, choice:2, score:3-5, choice:27+, score:6+, where choice:27+ is for head-only questions); * is used for any missing key.
enable_cuda_graphs() keeps extra buffers for its recordings.enable_cuda_graphs() needs a CUDA GPU. CPU and Apple Silicon (MPS) are untested.temperatures were fitted on.decide() takes one state per call; for bulk labeling, loop over the records (about 31 requests per second on one NVIDIA A100 80GB PCIe with enable_cuda_graphs()).์ฌ์ฉ ๊ฐ์ด๋ ์์ฝ
### ์ง๋ฌธ 1: contains_pii [์/์๋์ค (true/false)]์ฒ๋ผ ๊ทธ๋๋ก ๋ค์ด๊ฐ์, ์ง์๋ฌธ์ด ์์ผ๋ฉด ์ด๋ฆ์ด ๊ณง ์ง๋ฌธ์
๋๋ค. ์: q1 ๋์ contains_pii(๊ฐ์ธ์ ๋ณด๊ฐ ๋๋ฌ๋ ์๋), flag ๋์ safe_to_send(๊ทธ๋๋ก ๋ณด๋ด๋ ๋๋), cat ๋์ support_team(์ด๋ ํ์ด ๋งก๋), level ๋์ urgency(์ผ๋ง๋ ๊ธํ๊ฐ, 1โ5).none์ ๋ฃ์ต๋๋ค."type": "score")์ ์ ํ์ง๋ฅผ ๋ฎ์ ์ชฝ์์ ๋์ ์ชฝ ์์๋ก ์ ์ต๋๋ค. ์: ["1", "2", "3", "4", "5"], ["๋ฎ์", "๋ณดํต", "๋์"]. ๋ชจ๋ธ์ ์ด ์์๋ฅผ ์ฒ๋๋ก ์ฝ๊ณ , ๋ต์ expected๋ ์ด ์์์ ์๋ฒ(์ฒซ ์ ํ์ง = 1)์ผ๋ก ๊ณ์ฐํฉ๋๋ค.p_true 0.5 ์ด์์ ๋น ์ง์ผ๋ก ๋ณด๋ฉด ์ ๋ณด๊ฐ ์๋ 21๋ฌธํญ ์ค 8๊ฐ๋ฅผ ๋น ์ก๋ค๊ณ ํ๊ณ , 0.7 ์ด์์ด๋ฉด 5๊ฐ๋ก ์ค์ด๋ญ๋๋ค (์ ๋ง ๋น ์ง 30๋ฌธํญ ์ค ๋์น๋ ์๋ 2 โ 3).expected๋ ํ๋ฅ ๋ถํฌ๋ฅผ ๋ด
๋๋ค. held-out์์ 1๋ฑ ์ ํ์ง๊ฐ ์ ํํ ๋ง๋ ๊ฑด 57.7%์ง๋ง, ํ ์นธ ์ฐจ์ด ์์๋ 91.9%๊ฐ ๋ค์ด์ต๋๋ค.revision="v3.2"๋ก ๋ฒ์ ์ ๊ณ ์ ํฉ๋๋ค. ์ง์ ํ์ง ์์ผ๋ฉด ์ ๋ฒ์ ์ด ๋์ฌ ๋ ๋ฐ๋๋๋ค.temperatures๋ฅผ ์๊ธฐ ๋ฐ์ดํฐ ์๋ฐฑ ๊ฑด์ผ๋ก ๋ค์ ๋ง์ถฅ๋๋ค (model.config.temperatures).state and questions as decide() to POST https://api.blackmallangcow.com/v1/decide. Docs: https://api.blackmallangcow.com/docs/startdecide()์ ๊ฐ์ ์ํ์ ์ง๋ฌธ์ POST https://api.blackmallangcow.com/v1/decide๋ก ๋ณด๋ด๋ฉด ๋ฉ๋๋ค. ๋ฌธ์: https://api.blackmallangcow.com/docs/startenable_cuda_graphs() on one A100 80GB PCIe: 32 โ 32 msscore) questions on held-out: 57.0% โ 59.3% exactly right (90.9% โ 91.4% within one step), ECE 0.040 โ 0.020; unchanged on the eval set (50.5%, 99 questions)enable_cuda_graphs() on one A100 80GB PCIe: 26 โ 32 ms (the per-question chat turns add about 20 tokens per question)revision="v2"enable_cuda_graphs()revision="v1.1"revision="v1.0", e.g. AutoModel.from_pretrained("blackmallangcow/nari-decision-4b", revision="v1.0", trust_remote_code=True)Official validation/test splits of the training sources plus evaluation-only benchmarks (APEACH, Belebele ํ๊ตญ์ด, KoBBQ, MMMLU ํ๊ตญ์ด), at most 1,000 records per task. Never used for training or calibration.
| Questions | Accuracy (%) | ECE (raw โ calibrated) | NLL | Brier | Selective acc @80% (%) | |
|---|---|---|---|---|---|---|
| All | 79,688 | 85.6 | 0.009 โ 0.012 | 0.412 | 0.199 | 92.7 |
| Korean | 43,730 | 85.9 | 0.022 โ 0.025 | 0.407 | 0.209 | 92.3 |
| English | 35,958 | 85.3 | 0.012 โ 0.009 | 0.418 | 0.187 | 93.3 |
| noul | 45,371 | 90.6 | 0.005 โ 0.009 | 0.277 | 0.140 | 94.8 |
| choice | 27,907 | 83.9 | 0.028 โ 0.020 | 0.490 | 0.226 | 91.5 |
| score | 6,410 | 57.7 | 0.028 โ 0.026 | 1.025 | 0.502 | 62.7 |
The numbers above are from the evaluated model (LoRA kept separate). The released weights merge the LoRA into bf16; on a sample, accuracy of evaluated โ released: eval set: 79.83 โ 79.56 % (362 questions, answers agree 99.5%); held-out: 84.40 โ 84.40 % (2,602 questions, answers agree 99.7%).
Hand-checked scenarios close to real use: routing, guardrails, RAG relevance gates, agent branching, clarification. Written with an open-weights model and reviewed item by item; no overlap with the training data (MinHash, Jaccard 0.5).
| Questions | Accuracy (%) | ECE (raw โ calibrated) | NLL | Brier | Selective acc @80% (%) | |
|---|---|---|---|---|---|---|
| All | 519 | 80.5 | 0.045 โ 0.041 | 0.521 | 0.278 | 88.2 |
| Korean | 363 | 82.1 | 0.044 โ 0.048 | 0.490 | 0.262 | 89.3 |
| English | 156 | 76.9 | 0.095 โ 0.075 | 0.593 | 0.315 | 86.4 |
| noul | 220 | 85.5 | 0.050 โ 0.050 | 0.361 | 0.221 | 90.9 |
| choice | 200 | 86.5 | 0.061 โ 0.049 | 0.415 | 0.195 | 94.4 |
| score | 99 | 57.6 | 0.153 โ 0.118 | 1.090 | 0.572 | 60.0 |
| Category | Accuracy (%) | ECE |
|---|---|---|
| routing | 72.1 | 0.097 |
| guardrail | 83.0 | 0.057 |
| rag_gate | 83.5 | 0.066 |
| agent | 84.2 | 0.090 |
| clarify | 80.4 | 0.114 |
score) questionsExact accuracy is strict for ordinal labels, and most misses are one step off. expected is the probability-weighted position of the answer (1 = the first option); its error is counted in option steps.
| Set | Questions | Exact (%) | Within one step (%) | Mean error of expected (steps) |
ECE |
|---|---|---|---|---|---|
| Held-out | 6,410 | 57.7 | 91.9 | 0.62 | 0.026 |
| โณ HelpSteer2 response ratings, 0โ4 | 5,000 | 57.7 | 93.2 | 0.61 | 0.038 |
| โณ KLUE-STS similarity, 0โ5 | 519 | 56.6 | 97.1 | 0.53 | 0.075 |
| โณ Korean HateSpeech level (none < offensive < hate) | 471 | 64.8 | 97.5 | 0.40 | 0.125 |
| โณ HelpSteer2 preference strength, โ3 to 3 | 420 | 51.4 | 64.3 | 1.14 | 0.062 |
| System One eval set | 99 | 57.6 | 87.9 | 0.62 | 0.118 |
Each question is answered twice from the same forward pass: by the decision head and by the backbone's own answer letter. The released model averages the two (log-probabilities, 0.5 / 0.5) and then calibrates. Accuracy (%) / ECE, each readout calibrated on its own, on the questions that have an answer letter (at most 26 options): 75,688 held-out questions.
| Set | Decision head | Backbone letter | Averaged (this model) |
|---|---|---|---|
| Held-out | 85.6 / 0.016 | 83.8 / 0.030 | 85.7 / 0.011 |
| System One eval set | 80.9 / 0.059 | 77.8 / 0.041 | 80.5 / 0.041 |
Accuracy (%). The prompted backbone is Qwen/Qwen3.5-4B without training, asked one question per request with the options as letters (first-token letter probabilities). MMMLU-ko is OpenAI's professional Korean translation of MMLU (MIT); Belebele-ko is Meta's reading benchmark (CC BY-SA 4.0); the exams are the Ministry of Justice's 2026 rounds, after every round used for training (KOGL Type 1, source: ๋ฒ๋ฌด๋ถ).
| Benchmark | Questions | This model | Prompted Qwen3.5-4B | v2 |
|---|---|---|---|---|
| MMMLU-ko, all questions (knowledge) | 13,912 | 58.5 | 58.6 | 58.3 |
| Belebele-ko (reading) | 898 | 84.6 | 82.5 | 87.1 |
| Korean bar exam (15th) + legal ethics exam (17th), 2026 | 188 | 43.1 | 29.3 | โ |
Chance-corrected skill (0 = random guessing, 100 = perfect) on the public benchmark panel of the Decision Index leaderboard (edition 0.3, 140,178 scored requests), run locally with the leaderboard's own kit; unanswered requests count as wrong. The index is a fixed-weight mean of five areas. This is the public part only: the board's Full score adds private tests that its maintainers run. These are our own runs, not numbers taken from the leaderboard.
| v2 | This model | |
|---|---|---|
| Decision Index (public) | 38.49 | 44.80 |
| Knowledge & Reasoning | 24.6 | 28.8 |
| Language Understanding | 42.0 | 44.5 |
| Retrieval & Classification | 40.4 | 59.8 |
| Tools & Automation | 60.6 | 62.1 |
| Arts & Human Taste | 21.3 | 25.4 |
| Set | This model | Prompted Qwen3.5-4B | clef-flash (9B) | JEV-9B | laya-multilingual | laya (English) |
|---|---|---|---|---|---|---|
| Held-out | 85.6 / 0.012 | 66.0 / 0.038 | 72.8 / 0.014 | 61.1 / 0.040 | 50.5 / 0.025 | โ |
| Held-out, Korean | 85.9 / 0.025 | 68.8 / 0.040 | 75.2 / 0.021 | 67.3 / 0.028 | 51.0 / 0.029 | โ |
| Held-out, English | 85.3 / 0.009 | 62.6 / 0.047 | 69.8 / 0.027 | 53.2 / 0.059 | 49.9 / 0.040 | 52.1 / 0.030 |
| System One eval set | 80.5 / 0.041 | 66.9 / 0.052 | 73.2 / 0.077 | 73.4 / 0.088 | 34.5 / 0.119 | โ |
| Eval set, Korean | 82.1 / 0.048 | 67.8 / 0.061 | 73.8 / 0.077 | 76.0 / 0.115 | 36.4 / 0.110 | โ |
| Eval set, English | 76.9 / 0.075 | 64.7 / 0.058 | 71.8 / 0.098 | 67.3 / 0.072 | 30.1 / 0.173 | 36.5 / 0.153 |
5 requests from the System One eval set: 3 simple (2 questions, 43โ97-character state; 2 Korean, 1 English) and 2 complex (3 questions, 552โ900-character state; 1 Korean, 1 English). Each number is the mean over those requests of the median of 10 runs (after 3 warm-up runs). The other models were measured on one A100 80GB PCIe on 2026-10-15; this model (the released weights) was measured the same way on an A100 80GB PCIe on 2026-10-18. The computation is the same as v2's: 63.4 ms eager here, 64.3 ms for v2 on the first machine. Models run through transformers are eager (no compilation or CUDA graphs) unless the row says enable_cuda_graphs(); the vLLM servers use vLLM's default CUDA graphs.
| Model | Mean (ms) | Simple (ms) | Complex (ms) | How it runs |
|---|---|---|---|---|
| laya-multilingual | 22.7 | 22.1 | 23.5 | laya package (encoder), all questions in one pass |
| laya (English) | 30.4 | 27.0 | 35.5 | laya package (encoder), all questions in one pass |
| nari-decision-4b (this model) | 32.2 | 24.7 | 43.5 | transformers + enable_cuda_graphs(), all questions in one forward pass |
| nari-decision-4b (this model) | 63.4 | 62.7 | 64.5 | transformers, all questions in one forward pass |
| clef-flash (9B) | 73.4 | 66.4 | 83.8 | transformers, all questions in one forward pass |
| Prompted Qwen3.5-4B | 115.1 | 94.4 | 146.1 | local vLLM server, one request per question |
| JEV-9B | 141.4 | 85.7 | 224.9 | local vLLM server, one request per question |
Qwen/Qwen3.5-4B (text model only), frozen, with LoRA on every linear layer (attention, Gated DeltaNet projections, MLP). The LoRA is merged into the released weights.{"*": 1.0, "noul:2": 0.9278, "choice:3-5": 1.1189, "choice:6+": 0.9632, "choice:2": 1.0, "score:3-5": 1.0382, "choice:27+": 1.1189, "score:6+": 0.8937}| Source | Questions | License | Attribution |
|---|---|---|---|
| KLUE | 52,122 | CC BY-SA 4.0 | KLUE (Park et al., 2021), CC BY-SA 4.0 |
| KoSBiยทSQuARe (NAVER) | 32,001 | MIT | KoSBiยทSQuARe (Lee et al., 2023), NAVER, MIT |
| nari-decision synthetic decisions (D4) | 27,216 | Apache-2.0 (synthetic, generated and cross-checked by Apache-2.0 models) | Synthetic decisions generated with Qwen3.8-27B and cross-checked with Gemma-4-31B-it (Apache-2.0) |
| ABCD (Action-Based Conversations Dataset) | 16,000 | MIT | ABCD (Chen et al., 2021), ASAPP Research, MIT |
| MIRACL ํ๊ตญ์ด | 15,110 | Apache-2.0 | MIRACL (Zhang et al., 2023), Apache-2.0 |
| Schema-Guided Dialogue (SGD) | 14,000 | CC BY-SA 4.0 | Schema-Guided Dialogue (Rastogi et al., 2020), Google, CC BY-SA 4.0 |
| KorNLI ํ์ต ์ธํธ (SNLIยทMNLI ๊ธฐ๊ณ๋ฒ์ญ) | 14,000 | CC BY-SA 4.0 | KorNLI (Ham et al., 2020), Kakao Brain, CC BY-SA 4.0 |
| K-MHaS | 12,006 | CC BY-SA 4.0 | K-MHaS (Lee et al., 2022), CC BY-SA 4.0 |
| NoMIRACL ํ๊ตญ์ด | 12,003 | Apache-2.0 | NoMIRACL (Thakur et al., 2024), Apache-2.0 |
| MASSIVE 1.1 ํ๊ตญ์ด (ko-KR) | 12,000 | CC BY 4.0 | MASSIVE (FitzGerald et al., 2022), Amazon, CC BY 4.0 |
| HelpSteer2 (NVIDIA) | 12,000 | CC BY 4.0 | HelpSteer2 (Wang et al., 2024), NVIDIA, CC BY 4.0 |
| KoBEST (SKT) | 11,485 | CC BY-SA 4.0 | KoBEST (Kim et al., 2022), SKT, CC BY-SA 4.0 |
| MKQA ํ๊ตญ์ด | 10,000 | CC BY-SA 3.0 | MKQA (Longpre et al., 2021), Apple, CC BY-SA 3.0 |
| VitaminC | 10,000 | CC BY-SA 3.0 | VitaminC (Schuster et al., 2021), CC BY-SA 3.0 |
| CLINC150 (oos plus) | 10,000 | CC BY 3.0 | CLINC150 (Larson et al., 2019), CC BY 3.0 |
| HotpotQA (distractor) | 9,004 | CC BY-SA 4.0 | HotpotQA (Yang et al., 2018), CC BY-SA 4.0 |
| 3i4K | 8,000 | CC BY-SA 4.0 | 3i4K (Cho et al., 2018), CC BY-SA 4.0 |
| Civil Comments (TFDS/HF) | 7,000 | CC0 1.0 | Civil Comments (Borkan et al., 2019), Jigsaw, CC0 |
| ์ง์์ฒด ์๋ฆผ ๋ฌธ์์ ๋ถ์ฐ ์ฌ๋๋ฌธ์ | 6,161 | ๊ณต๊ณต๋ฐ์ดํฐ ์ด์ฉํ๋ฝ๋ฒ์ ์ ํ ์์ | ํํยท์์ยท๊ตฐํฌยท์ฉ์ธยท์์ยท๋จ์์ฃผยท๋๋์ฒยท๊ณผ์ฒยท๊ธ์ฒ๊ตฌ ์๋ฆผ ๋ฌธ์, ๋ถ์ฐ๊ด์ญ์ ์ฌ๋๋ฌธ์, ๊ณต๊ณต๋ฐ์ดํฐํฌํธ |
| Aegis AI Content Safety 1.0 (NVIDIA) | 6,001 | CC BY 4.0 | Aegis 1.0 (Ghosh et al., 2024), NVIDIA, CC BY 4.0 |
| Amazon ESCI ํ์ต ๋ถํ | 6,000 | Apache-2.0 | Shopping Queries Dataset (Reddy et al., 2022), Amazon, Apache-2.0 |
| ShARC | 6,000 | CC BY-SA 3.0 | ShARC (Saeidi et al., 2018), CC BY-SA 3.0 |
| STAR | 6,000 | MIT | STAR (Mosig et al., 2020), Rasa, MIT |
| PAWS-X ํ๊ตญ์ด | 6,000 | Free use, attribution requested (Google) | PAWS-X (Yang et al., 2019), Google |
| WinoGrande XL ํ์ต ๋ถํ | 6,000 | CC BY | WinoGrande (Sakaguchi et al., 2020), AI2, CC BY |
| Dynamically Generated Hate Speech (DynaHate) v0.2.3 | 6,000 | CC BY 4.0 | DynaHate (Vidgen et al., 2021), CC BY 4.0 |
| HoVer ํ์ต ๋ถํ | 6,000 | CC BY-SA 4.0, ์ฝ๋ MIT | HoVer (Jiang et al., 2020), CC BY-SA 4.0 |
| Korean HateSpeech (kocohub / BEEP!) | 6,000 | CC BY-SA 4.0 | Korean HateSpeech Dataset (Moon et al., 2020), CC BY-SA 4.0 |
| GSM8K (OpenAI) | 5,877 | MIT | GSM8K (Cobbe et al., 2021), OpenAI, MIT |
| SQuAD 2.0 | 5,000 | CC BY-SA 4.0 | SQuAD 2.0 (Rajpurkar et al., 2018), CC BY-SA 4.0 |
| FaithDial | 5,000 | MIT | FaithDial (Dziri et al., 2022), MIT |
| MASSIVE 1.1 ์์ด (en-US, SLURP ์๋ฌธ) | 5,000 | CC BY 4.0 | MASSIVE (FitzGerald et al., 2022), Amazon, CC BY 4.0 |
| AQuA-RAT | 4,000 | Apache-2.0 | AQuA-RAT (Ling et al., 2017), Google DeepMind, Apache-2.0 |
| CommonsenseQA 2.0 | 4,000 | CC BY 4.0 | CommonsenseQA 2.0 (Talmor et al., 2021), AI2, CC BY 4.0 |
| CommonsenseQA 1.11 | 4,000 | MIT | CommonsenseQA (Talmor et al., 2019), MIT |
| Social IQa | 4,000 | CC BY 4.0 | Social IQa (Sap et al., 2019), AI2, CC BY 4.0 |
| Taskmaster-3 | 4,000 | CC BY 4.0 | Taskmaster-3 (Byrne et al., 2020), Google, CC BY 4.0 |
| MASSIVE-Agents ํจ์ ์ ์ 55๊ฐ | 4,000 | CC BY 4.0 | MASSIVE-Agents, Amazon, CC BY 4.0 |
| Fool Me Twice (FM2) | 4,000 | Apache-2.0 | Fool Me Twice (Eisenschlos et al., 2021), Google, Apache-2.0 |
| BLiMP | 4,000 | CC BY | BLiMP (Warstadt et al., 2020), CC BY |
| BANKING77 ํ์ต ๋ถํ | 3,000 | CC BY 4.0 | BANKING77 (Casanueva et al., 2020), PolyAI, CC BY 4.0 |
| UCI SMS Spam Collection | 3,000 | CC BY 4.0 | SMS Spam Collection (Almeida & Gรณmez Hidalgo, 2011), UCI, CC BY 4.0 |
| PAWS (Wiki, labeled_final) | 3,000 | Free use, attribution requested (Google) | PAWS (Zhang et al., 2019), Google |
| QASC | 3,000 | CC BY 4.0 | QASC (Khot et al., 2020), AI2, CC BY 4.0 |
| ํ๊ตญ์ด ์ง๋ฌธ ์ (Question_pair) | 3,000 | MIT | Question_pair (songys), MIT |
| MKQA ์์ด | 3,000 | CC BY-SA 3.0 | MKQA (Longpre et al., 2021), Apple, CC BY-SA 3.0 |
| Nazario ํผ์ฑ ๋ฉ์ผ ๋ชจ์ (2004~2025 ๋ฉ์ผํจ) | 3,000 | CC BY 4.0 | Phishing corpus by Jose Nazario, CC BY 4.0 |
| StrategyQA | 2,095 | MIT | StrategyQA (Geva et al., 2021), MIT |
| BoolQ | 2,000 | CC BY-SA 3.0 | BoolQ (Clark et al., 2019), Google, CC BY-SA 3.0 |
| CLAIR ์ฌ๊ธฐ ๋ฉ์ผ ๋ชจ์ | 1,947 | CC BY-SA 3.0 US | CLAIR collection of fraud email (Radev, 2008), CC BY-SA 3.0 |
| ๋ณํธ์ฌ์ํ ๊ธฐ์ถยท์ ํํ ์ต์ข ์ ๋ต (๋ฒ๋ฌด๋ถ) | 1,824 | KOGL Type 1 (attribution) | ๋ฒ๋ฌด๋ถ, ๋ณํธ์ฌ์ํ ๊ธฐ์ถ๋ฌธ์ (๊ณต๊ณต๋๋ฆฌ ์ 1์ ํ) |
| SIB-200 ์์ด (eng_Latn) | 648 | CC BY-SA 4.0 | SIB-200 (Adelani et al., 2024), CC BY-SA 4.0 |
| ๋ฒ์กฐ์ค๋ฆฌ์ํ ๊ธฐ์ถยท์ต์ข ์ ๋ต (๋ฒ๋ฌด๋ถ) | 574 | KOGL Type 1 (attribution) | ๋ฒ๋ฌด๋ถ, ๋ฒ์กฐ์ค๋ฆฌ์ํ ๊ธฐ์ถ๋ฌธ์ (๊ณต๊ณต๋๋ฆฌ ์ 1์ ํ) |
| SIB-200 ํ๊ตญ์ด | 84 | CC BY-SA 4.0 | SIB-200 (Adelani et al., 2024), CC BY-SA 4.0 |
score) questions are the weakest type: 57.7% exactly right on held-out (91.9% within one step). Most of them are human ratings (HelpSteer2), where people also differ by a step. On the System One eval set they are poorly calibrated (ECE 0.118 on 99 questions).Apache-2.0 for the weights and code. The training data includes CC BY and CC BY-SA sources; their attribution is listed in the table above.
The Korean national exam questions are public works of the Ministry of Justice released under KOGL Type 1 (๊ณต๊ณต๋๋ฆฌ ์ 1์ ํ, attribution required); the attribution is in the table above.
Qwen/Qwen3.5-4B