KnowLine-4B-Gen1

A 4B System One decision model from PelaAI, trained on a single all-in-one machine by an agent-driven data loop.

Inference guide · 中文说明 · weights Apache-2.0

Newer versions: KnowLine-4B-Gen3 (Decision Index 0.3 public score 63.11) and KnowLine-4B-Gen2 (62.54) are out. Both are at or above this model on our held-out, C-Eval, Open-Jev and prompt-injection evaluations.

Highlights:

• Decision Index 0.3, public suite: 60.47 (self-run). For comparison: Jev 1.13 scores 57.96 (board), and the best ≤5B model on the board has a public score of 50.82; see Comparison.
• An AI agent ran the whole loop. In each round it:
  • found the areas where the model was weak;
  • proposed a targeted data group;
  • built and decontaminated the data;
  • trained and evaluated on it;
  • kept or rejected the change based on the evidence.
• KOF '98 harness: 17 wins, 0 losses, 1 draw against Jev 1.13 in a single-bout mirror round robin. For the play style, see Game harness.

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

• one forward pass per question, with no generated text;
• the output is the probability of each option's label token;
• existing Jev clients only need a new base URL.

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-4B-Gen1 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

• Without SGLang: python knowline_server.py --model PelaAI/KnowLine-4B-Gen1 --backend hf --port 8080 uses transformers only. It is slower and runs bf16.
• Front end: knowline_server.py is a single file and needs only transformers and requests.
• Full settings: the exact settings of our Decision Index runs are in INFERENCE.md.

Comparison

Decision Index 0.3, public suite. The full 0.3 score adds private tests that only the maintainers run (0.5 same-skill, 0.3 new-domain); ours is not available yet.

Our public-suite scores by area:

Releases

The model is trained in a self-evolving loop, and new versions will follow.

Evaluation

All results are self-run and not verified by a third party.

Held-out and Chinese evaluations

Web operation (Mind2Web official test splits, evaluation only)

• So far this is only used to explore the model in RPA-style automation and to check that it generalises to some degree.
• The task is to pick the target element among it and up to 5 other candidates sampled from the page. This is easier than the original Mind2Web protocol, so do not compare it with the Mind2Web leaderboard.

Game harness (KOF '98)

• Setup: single-bout character-mirror round robin, 11 players, 18 games per pair (990 games), argmax actions.
• Result: win rate 0.883 [0.83, 0.92], Elo 1910, tied for first of 11 with another internal checkpoint.
• Head-to-head: 17-0-1 against Jev 1.13, 15-3 against StartLux-Decision-4B, 18-0 against Clef-Flash.
• Caveat: the policy relies heavily on one move, a 623C anti-air uppercut used about 55% of the time. This is a measured result in this harness, not general fighting-game skill.

Disclosures

• Decision Index training data: about 25% of the data is in Decision Index format, that is, synthetic items written in the benchmarks' request formats. No Decision Index test item is included.
• Decontamination:
  • Every component text (state, instructions, option texts) was checked against every Decision Index row and all of our evaluation sets.
  • Rows that are identical, or share any run of 50 consecutive characters, were removed.
• Game data: labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation states.
• No model outputs as labels: no output of Jev or any other decision model was used as a training label.
• Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but not all were checked by a human.

Limitations

• Knowledge-heavy reasoning: weaker than larger models. The knowledge area is 37.7 (0.3); MMLU-Pro, GPQA and HLE are close to the base model.
• Math: answered without reasoning. The rebuilt 0.3 GSM8K scores 29.5.
• Prompt injection: an instruction planted in the state changes the answer about 9% of the time on our injection set. Keep untrusted text clearly delimited.
• Calibration: ECE is about 0.05-0.06 on choice and yes/no questions and about 0.12 on score questions. For score questions, fit a temperature on your own data. On the Decision Index 0.3 public suite, computed the board's way, ECE is about 0.08-0.09 and 3.3% of answers are wrong at ≥95% confidence (board median 1.3%): the model is overconfident overall.
• Private tests: part of our public-suite advantage comes from adapting to the question formats, so the 0.3 private tests may score lower.

Citation

@misc{knowline4bgen1,
  title  = {KnowLine-4B-Gen1: a 4B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-4B-Gen1}
}

License

• Weights: Apache-2.0, the same as the base model.
• Code: knowline_server.py is MIT (see the file header).