KnowLine-26B-A4B-Gen2

The second release of PelaAI's System One decision model on a Gemma 4 base: a 26B-A4B mixture of experts, trained in the same agent-driven data loop as KnowLine-35B-A3B and the KnowLine-4B line.

Inference guide · 中文说明 · weights Apache-2.0 · siblings: KnowLine-26B-A4B-Gen1, KnowLine-35B-A3B-Gen1, KnowLine-4B-Gen4

Highlights:

• Decision Index 0.3 public suite: 66.94 (self-run), up from 65.76 for KnowLine-26B-A4B-Gen1. KnowLine-35B-A3B-Gen1 scores 69.49 on the same suite and 62.92 on the board's full score; see Comparison.

• Better on Open-Jev 1.1: test / OOD 88.5 / 87.9, up from 83.2 / 85.0 for Gen1 (35B-A3B-Gen1: 91.0 / 90.4). The gain on OOD is significant.

• Decision tasks from new domains: on our larger set of decision tasks from domains not in training (34 tasks) it scores 60.8 (Gen1: 59.9; 35B-A3B-Gen1: 52.7). On our older 10-task set it scores 55.8 (Gen1: 55.2; 35B-A3B-Gen1: 56.4).

• Harder to steer with planted instructions: on our injection set, 0.1% of answers changed because of an instruction planted in the state; with injection templates not seen in training, 1.0% did (Gen1: 0.3% and 1.8%; 35B-A3B-Gen1: 0.0% and 0.8%).

• Calibrated probabilities: a temperature is folded into the weights. Computed with the board's method, ECE is about 0.01 (board median: 0.078) and the Brier score about 0.26-0.28 (board median: 0.46); see Calibration.

• Our other evaluations: English in-house set 70.6 (Gen1: 70.6), Chinese in-house sets 72.6 / 69.8 (72.2 / 69.2), C-Eval 74.8 (74.3), JevBench / JevBench-hard 88.3 / 76.6 (89.6 / 80.2). It is still clearly lower than the 35B-A3B on Chinese (C-Eval 87.6); see Evaluation.

• Images without any image training: trained on text only, it keeps the base model's vision tower. On our reconstruction of the Decision Index 0.3.1 vision public suite it scores 64.7 (Gen1: 64.9; stock gemma-4-26B-A4B-it: 59.7; 35B-A3B-Gen1: 71.2); see Images.

• KOF '98 harness: 16-2-0 against Jev, 17-1-0 against StartLux-Decision-35B-A3B and 6-6-6 against KnowLine-35B-A3B-Gen1 in single-bout mirror matches; see Game harness.

• An AI agent ran the whole loop end to end. In each round it:

  • identified where the model was weak;
  • proposed a targeted data group;
  • built and decontaminated the data;
  • trained and evaluated the model on it;
  • kept or rejected the change based on the evidence.

  For this model, see How Gen2 was made.

KnowLine is an independent model. It is not affiliated with or endorsed by TypeSafe, Jev or Google, and it is not derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

• one forward pass per question, with no text generation;
• the output is the probability of each option's label token;
• existing Jev clients only need to change the base URL, and users of other KnowLine models only need to change the model name, because the interface is the same.

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-26B-A4B-Gen2 0 8080     # SGLang (bf16) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

• Front end: knowline_server.py is a single file that depends only on transformers and requests. It is the same code as in the other KnowLine releases; it renders each question with the model's own chat template.
• Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.

How Gen2 was made

1. What to fix. Gen1 was level with KnowLine-35B-A3B-Gen1 on our English in-house set and ahead on our 34-task set of new-domain tasks, but significantly lower on our Chinese in-house sets and on Open-Jev 1.1 OOD. The agent checked whether this gap comes from the base model or from data coverage.

2. Data. A third round on top of Gen1:

   • rows from the same pool of public datasets and synthetic decision tasks that were not used in Gen1's rounds, across all languages and task families;
   • a replay share of Gen1's first round and the injection-invariance items of its second round.

   All data went through the same decontamination check as the earlier releases; see Disclosures.

3. Training. One epoch on the third round with a LoRA adapter, merged into the Gen1 weights. The readout temperature is the same as for Gen1 (T = 1.25); on our held-out sets it gives a lower calibration error than the value refitted for Gen2. It changes the size of the probabilities, but not which option ranks first.

4. Selection. Against Gen1, Open-Jev 1.1 OOD rose significantly (85.0 → 87.9); our English and Chinese in-house sets, JevBench-hard and new-domain tasks show no significant drop; the injection rate fell (0.3% → 0.1%; unseen templates 1.8% → 1.0%). So the gap was mostly data coverage, and this model is the release. Against KnowLine-35B-A3B-Gen1 it is still lower on Chinese and on Open-Jev 1.1 OOD.

Comparison

Scores on the Decision Index 0.3 board. The full score also includes private tests that only the maintainers run (weighted 0.5 same-skill and 0.3 new-domain); the full score of this model is not available yet.

Board rows as of the board's latest update (2026-10-16).

Public-suite scores by area (DI 0.3):

Releases

The model is trained in a self-evolving loop, and new versions will follow.

Evaluation

All results are self-run and have not been verified by a third party.

Held-out and Chinese evaluations

In a paired sign test against Gen1, the gain on Open-Jev 1.1 OOD is significant (p < 0.001); the differences on the English and Chinese in-house sets (p = 1.0, 0.44 and 0.43) and on JevBench-hard (80.2 → 76.6, p = 0.34) are not. Against KnowLine-35B-A3B-Gen1, it remains lower on C-Eval and on Open-Jev 1.1.

Web operation (Mind2Web official test splits, evaluation only)

• Element selection is 1.1-2.4 points higher than Gen1 on all three splits (87.1 / 87.0 / 87.0); operation is within 0.6 points of Gen1. Still lower than 35B-A3B-Gen1 on all three splits.
• The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web leaderboard.

Images (zero-shot)

• No image training: Gen2 keeps the base model's vision tower unchanged and saw no images in training. knowline_server.py accepts OpenAI-style image / image_url content parts in a chat state and passes them to SGLang; each question is still one prefill.
• How we measured it: the same reconstruction of the Decision Index 0.3.1 vision public suite as for the earlier releases (11 benchmarks, chance-corrected skill and the board's weights), run through the release server, 0 failed requests. We also ran the official vision public suite (12,210 rows, complete).

• Level with Gen1, below the 35B-A3B: within 0.2 points of Gen1 on both suites (Gen1 official suite: 63.4), and 6.5 points lower than KnowLine-35B-A3B-Gen1 on our reconstruction; the stock gemma-4-26B-A4B-it scores 59.7 there.
• Indicative only: the reconstruction is ours, not the board's rows; the official-suite column is a self-run of the official kit.

Game harness (KOF '98)

• Setup: single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for the other KnowLine releases.
• Result: 16-2-0 against Jev, 17-1-0 against StartLux-Decision-35B-A3B and 6-6-6 against KnowLine-35B-A3B-Gen1; 39-9-6 overall (Gen1: 15-2-1, 17-1-0 and 8-9-1).
• How the games ended: against Jev, 9 of the 16 wins were by K.O. and 7 at time-out; against StartLux-Decision-35B-A3B, 13 of 17 by K.O.; against KnowLine-35B-A3B-Gen1, 5 of 6 wins and 4 of 6 losses by K.O., and all 6 draws at time-out.
• Play style: special_2 46%, heavy kick 31%, heavy punch 9%, special_3 6%, wait 3%.
• Caveats: 18 games per pair give wide intervals; this is a measured result in this harness, not general fighting-game skill.

Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are used.

• Checked against the board: for KnowLine-35B-A3B-Gen1, the board's own figures (ECE 0.048, Brier 0.261) fall inside the ranges our computation gave, so we use the same computation here.
• Temperature in the weights: the final norm weight and the final logit soft-cap include a readout temperature T = 1.25, the same as for Gen1, so every final logit is divided by 1.25 before the softmax over the option labels. The top-ranked option does not change; the probabilities are just less extreme. On our held-out sets, ECE with the temperature in the weights is 0.017 (Gen1: 0.037; 35B-A3B-Gen1: 0.025).
• Why ranges: the board does not publish which 32 benchmarks it uses, so we report ranges over the plausible benchmark sets, as for the earlier releases.
• Recommendation: the temperature was fitted on our data. If your use depends on probability thresholds, check calibration on your own data.

Disclosures

• Decision Index format training data: in the first round, about 34% of the rows target Decision Index benchmarks in their request format or a reworded version of it: train splits of public datasets rewritten in that format (for example WinoGrande XL, ACOS and GSM8K) and synthetic items written in the same format. Another 0.5% are copies of the same items rendered in other formats. The second round is one third injection-invariance items and two thirds a replay of the first round. In the third round, about 26% of the rows target Decision Index benchmarks in the same way, plus 3.1% format copies.
• Decontamination: all training data was checked with the same tool as the earlier releases against every Decision Index 0.3 row, all of our evaluation sets and the other public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM). In the third round, 751 rows that nearly duplicated an evaluation item, repeated its question or claim, or contained its whole state were removed (in Gen1's rounds: 491); the replayed parts come from data that had already passed the same check.
• Selection on evaluations: we compared third-round candidates with Gen1 on our release checks; the release is the candidate that passed all of them. It is still below KnowLine-35B-A3B-Gen1 on Chinese and Open-Jev 1.1 OOD (see Evaluation).
• No model outputs as labels: no output of Jev or any other decision model was used as a training label.
• Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but not all of them were checked by a human.

Limitations

• Chinese: better than Gen1 on Open-Jev 1.1, but still clearly lower than KnowLine-35B-A3B-Gen1 on C-Eval (74.8 against 87.6). For Chinese-heavy use, prefer the 35B-A3B.
• JevBench-hard: 76.6 against 80.2 for Gen1; the drop is not significant, but check it if your use is close to JevBench-hard.
• Size and speed: about 52 GB of bf16 weights, so it needs one GPU with 80 GB or more. One request at a time on one H20, requests take 43.1 ms median but 237.6 ms at the 95th percentile, a longer tail than KnowLine-35B-A3B-Gen1 with FP8 serving (174 ms); see INFERENCE.md.
• Web operation and images: below KnowLine-35B-A3B-Gen1 on our Mind2Web and vision evaluations; there was no image training, so check image use on your own data.
• Prompt injection: planted instructions changed 0.1% of answers on our injection set and 1.0% with unseen templates. Other attacks may still work, so keep untrusted text clearly delimited.
• Private tests: part of the public-suite score comes from benchmarks whose tasks are in our training data, so the 0.3 private tests may show a lower score.

Citation

@misc{knowline26ba4bgen2,
  title  = {KnowLine-26B-A4B-Gen2: a 26B-A4B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-26B-A4B-Gen2}
}

License

• Weights: Apache-2.0, the same as the base model.
• Code: knowline_server.py is MIT (see the file header).