KnowLine-4B-Gen5

The fifth release of PelaAI's 4B System One decision model, trained on a single all-in-one machine through an agent-driven data loop. This round targets generalization rather than the leaderboard.

Inference guide · 中文说明 · weights Apache-2.0 · previous version: KnowLine-4B-Gen4

Highlights:

• Less fitted to the leaderboard. We noticed that our earlier models may have focused too much on the leaderboard: on evaluation sets we built ourselves to test generalization, they did clearly worse than their public score suggests. For Gen5 we adjusted the data mix and retrained; see What changed in Gen5.

• Better on decision tasks from new domains: on our 18 real-text decision tasks from domains not in training (built after Gen4) Gen5 scores 51.3, up from 46.1 for Gen4; on our 34-task set it scores 43.0, up from 39.0.

• Better on the same skills with different data: on our sibling sets, which test the skill of a public benchmark on other data, Gen5 averages 63.7, up from 60.7 for Gen4, and it is significantly better on our Simplified and Traditional Chinese in-house sets.

• Decision Index 0.3 public suite: 65.67 (self-run; Gen4: 64.90). The public score was not what we optimized; see Comparison.

• Harder to steer with planted instructions: 0.6% of answers changed because of an instruction planted in the state (Gen4: 1.6%); with injection templates not seen in training, 2.5% (Gen4: 6.2%).

• Calibrated probabilities: a temperature is folded into the weights. Computed with the board's method, ECE is about 0.04-0.05 and the Brier score about 0.28-0.31; see Calibration.

• Images without any image training: 66.0 on our reconstruction of the Decision Index 0.3.1 vision public suite (Gen4: 66.4; stock Qwen3.5-4B: 59.4); see Images.

• KOF '98 harness: 14-3-1 against Jev, 14-3-1 against StartLux-Decision-4B and 9-9-0 against Gen4 in single-bout mirror matches; see Game harness.

• An AI agent ran the whole loop end to end. In each round it:

  • identified where the model was weak;
  • proposed a targeted data group;
  • built and decontaminated the data;
  • trained and evaluated the model on it;
  • kept or rejected the change based on the evidence.

  For this round, see What changed in Gen5.

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

• one forward pass per question, with no text generation;
• the output is the probability of each option's label token;
• existing Jev clients only need to change the base URL, and users of earlier versions only need to change the model name, because the interface and serving settings are unchanged.

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-4B-Gen5 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

• Front end: knowline_server.py is a single file that depends only on transformers and requests. It is the same file as in Gen4.
• Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.

What changed in Gen5

1. What we noticed. The Decision Index public suite is only part of the picture: the board's full score comes from private tests of the same skills on new data and from tasks in new domains. To see how well our models generalize, we built evaluation sets of the same kind: decision tasks from domains absent from training (a 34-task set, and an 18-task set of real-text decisions such as refunds, utility bills, appeals, eligibility checks, multilingual requests and legal questions), and sibling sets that test the skill of a public benchmark on different data. On these, our earlier models did clearly worse than their public scores suggested. For example, removing the training rows written in the format of Decision Index benchmarks cost about 20 public points but changed our new-domain score very little. The board shows the same pattern for our KnowLine-35B-A3B-Gen1, which ranks 7th on the public suite but 27th on the full score of Decision Index 0.4.
2. New data mix. We rebalanced the training data toward judgements over real, ambiguous text: fewer verbatim copies of items in benchmark format and more reworded layouts; new real-text data on business policies, several languages, legal and contract questions, tables, human judgement and syllogisms; and a smaller share of procedural items (rule execution, arithmetic, logic puzzles). All of the data went through our decontamination check; see Disclosures.
3. Training. A model was retrained from the base Qwen3.5-4B on the new mix, for one epoch. Gen5 is the 50/50 weight average of this model and a second model, which continues our earlier 4B line on a further round of data (rule-following and routing tasks, agent decisions, reasoning items, and a replay of earlier data). A single temperature was then fitted and folded into the weights. It changes the size of the probabilities, but not which option ranks first.
4. Selection. We compared the candidates on our estimate of the board's full score (built from our own evaluation sets for the private same-skill and new-domain parts) and on our new-domain sets, not on the public score. Gen5 had the highest estimate and the highest new-domain scores among them.
5. What it means. The public score moved little (64.90 → 65.67): it was not the target. The gains are on the tests that measure transfer: our new-domain sets (+5.2 on the 18-task set, +4.0 on the 34-task set), our sibling sets (+3.0), our Chinese in-house sets (+1.6 and +1.8, both significant) and unseen injection templates (6.2% → 2.5%).

Comparison

Scores on the Decision Index. Since Decision Index 0.4 (2026-10-19), the board's full score is 60% private tests of the same skills and 40% private tasks from new domains; the public suite is shown but no longer counts. Our full score is not available yet.

Board rows as of the board's latest update (Decision Index 0.4).

Public-suite scores by area (DI 0.3):

Releases

The model is trained in a self-evolving loop, and new versions will follow.

Evaluation

All results are self-run and have not been verified by a third party.

Held-out, Chinese and generalization evaluations

In a paired sign test against Gen4, the gains on the Simplified and Traditional Chinese in-house sets (p = 0.004 and p = 0.028) and on unseen injection templates (p < 0.001) are significant. The differences on the English in-house set (p = 0.90), Open-Jev 1.1 OOD (p = 0.23) and JevBench-hard (72.1 → 66.7, p = 0.15) are not.

Web operation (Mind2Web official test splits, evaluation only)

• About the same as Gen4; the differences are within noise.
• The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web leaderboard.

Images (zero-shot)

• No image training: Gen5 keeps the base model's vision tower unchanged and saw no images in training. knowline_server.py accepts OpenAI-style image / image_url content parts in a chat state and passes them to SGLang; each question is still one prefill.
• How we measured it: the same reconstruction of the Decision Index 0.3.1 vision public suite as for Gen4 (11 benchmarks, chance-corrected skill and the board's weights), run through the release server.

• About the same as Gen4, and 6.6 points above the stock model.
• Indicative only: these are self-run numbers on our reconstruction, not on the board's rows.

Game harness (KOF '98)

• Setup: single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for Gen1-Gen4.
• Result: 14-3-1 against Jev, 14-3-1 against StartLux-Decision-4B and 9-9-0 against Gen4; 37-15-2 overall.
• How the games ended: against Jev, 2 of the 14 wins were by K.O. and 12 at time-out; against StartLux-Decision-4B, 6 of 14 by K.O. All 6 losses to these two came at time-out. Against Gen4, 4 of the 9 wins and 4 of the 9 losses were by K.O.
• Play style: special_2 47%, special_1 26%, heavy kick 11%, heavy punch 7%, special_3 4%.
• Caveats: 18 games per pair give wide intervals; this is a measured result in this harness, not general fighting-game skill.

Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are used.

• Temperature in the weights: the final norm weight includes a readout temperature T = 1.50, so every logit is divided by 1.50 before the softmax over the option labels. The top-ranked option does not change; the probabilities are just less extreme. T was fitted on a set of our own evaluation items whose distribution is covered by the training data, not on the held-out sets or on Decision Index rows. On our held-out sets, ECE with the temperature in the weights is 0.027 (Gen4: 0.043).
• Why ranges: the board does not publish which 32 benchmarks it uses, so we report ranges over the plausible benchmark sets, as for the earlier releases.
• Recommendation: the temperature was fitted on our data. If your use depends on probability thresholds, check calibration on your own data.

Disclosures

• Decision Index format training data: Gen5 combines two training lines. In the data of this round, about 24% of the rows target Decision Index benchmarks in their request format or a reworded version of it, and another 1.4% are copies of the same items rendered in other formats. These include train splits of public datasets rewritten in that format (for example GSM8K, WinoGrande XL, ACOS and Amazon ESCI) and synthetic items written in the same format.
• Decontamination: all training data of this round, new and replayed, was checked with the same tool as Gen4 against every Decision Index 0.3 row, all of our evaluation sets (including the new-domain and sibling sets above) and the other public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM). About 22,000 of 3.1 million rows were removed: rows that nearly duplicate an evaluation item, repeat its question or claim, or contain its whole state, and, in a stricter second pass, most rows that share a source passage with an evaluation item.
• Selection on evaluations: we picked Gen5 by our estimate of the board's full score and by our new-domain sets, not by the public score.
• Game data: labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation states.
• No model outputs as labels: no output of Jev or any other decision model was used as a training label.
• Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but not all of them were checked by a human.

Limitations

• Knowledge-heavy reasoning: weaker than larger models. The knowledge area score is 45.0 (DI 0.3); a 4B model is limited here, and the new-domain part of the board favours larger models.
• JevBench-hard: 66.7 against 72.1 for Gen4; the drop is not significant, but check it if your use is close to JevBench-hard.
• New domains: better than Gen4 on our own sets, but still well below larger models; our sets are our own construction and not the board's private tasks.
• Prompt injection: an instruction planted in the state changes the answer about 0.6% of the time on our injection set, and about 2.5% of the time with templates not seen in training. Keep untrusted text clearly delimited.
• Images: there was no image training. The image scores above are zero-shot and self-run on our own reconstruction of the suite, so check them on your own data before relying on them.

Citation

@misc{knowline4bgen5,
  title  = {KnowLine-4B-Gen5: a 4B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-4B-Gen5}
}

License

• Weights: Apache-2.0, the same as the base model.
• Code: knowline_server.py is MIT (see the file header).