KnowLine-35B-A3B-Gen1

The first release of PelaAI's 35B-A3B System One decision model, trained on the same single all-in-one machine and agent-driven data loop as the KnowLine-4B line.

Inference guide · 中文说明 · weights Apache-2.0 · smaller sibling: KnowLine-4B-Gen4

Highlights:

• Decision Index 0.3 public suite: 69.49 (self-run), 4.59 points above KnowLine-4B-Gen4 (64.90). On the board, the highest public score is 65.64 (RSI-Jev v6.1 VL 27B), followed by vLLM-SR d3 (27B) at 65.45, which has the highest full score (66.26); Perplexity Decider v1.1 (27B) scores 62.25 and Jev 1.13 scores 57.96; see Comparison.

• Stronger where the 4B was weakest: knowledge and reasoning rises from 44.6 (Gen4) to 52.5, close to Jev 1.13 (53.9). The largest gains over Gen4 are on knowledge-heavy benchmarks such as MMLU-Pro (+15.0) and BBH (+11.8); see Where the gain over Gen4 comes from.

• Better on new domains: on our own decision tasks from domains not in training it scores 56.4, against 49.3 for Gen4.

• Harder to steer with planted instructions: on our injection set, no answer changed because of an instruction planted in the state (Gen4: 1.6%); with injection templates not seen in training, 0.8% did (Gen4: 6.2%).

• Calibrated probabilities: a temperature is folded into the weights. Computed with the board's method, ECE is about 0.04-0.05 (board median: 0.078) and the Brier score about 0.25-0.27, below the lowest Brier score on the board (0.28); see Calibration.

• Our other evaluations: Open-Jev 1.1 test / OOD 91.0 / 90.4 (Gen4: 88.4 / 87.1), C-Eval 87.6 (78.7), and higher scores on our Chinese held-out sets. JevBench / JevBench-hard 85.7 / 70.3 (86.1 / 72.1): slightly lower (the JevBench-hard difference is not significant).

• Images without any image training: like Gen4, it was trained on text only and keeps the base model's vision tower. On our reconstruction of the Decision Index 0.3.1 vision public suite it scores 71.2, against 67.8 for the stock Qwen3.5-35B-A3B on the same harness (self-run), with gains on some benchmarks and small drops on others; see Images.

• KOF '98 harness: 17-0-1 against Jev 1.13, 18-0-0 against StartLux-Decision-35B-A3B and 12-4-2 against KnowLine-4B-Gen4 in single-bout mirror matches; see Game harness.

• An AI agent ran the whole loop end to end. In each round it:

  • identified where the model was weak;
  • proposed a targeted data group;
  • built and decontaminated the data;
  • trained and evaluated the model on it;
  • kept or rejected the change based on the evidence.

  For this model, see How Gen1 was made.

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

• one forward pass per question, with no text generation;
• the output is the probability of each option's label token;
• existing Jev clients only need to change the base URL, and users of KnowLine-4B only need to change the model name, because the interface is the same.

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-35B-A3B-Gen1 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

• Front end: knowline_server.py is a single file that depends only on transformers and requests. It is the same code as in KnowLine-4B-Gen4.
• Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.

How Gen1 was made

1. Why a larger base? The team trained the KnowLine-4B series, most recently updated to KnowLine-4B-Gen4. Over four rounds, it gained most on retrieval, tools and taste, while knowledge and reasoning stayed the weakest area (44.6 in Gen4). The agent trained the same data on a larger base to see how much of that gap comes from model size.

2. Data. Two rounds:

   • all training data of the KnowLine-4B line up to Gen4, in one set;
   • a second round of inputs no earlier round used: more rows from the same public train sets, format variants of existing items, about 40 new task families from domains not covered before, and synthetic scenarios, plus a replay share of the first round (arithmetic, knowledge, Jev-style and injection-invariance items at a higher rate).

   All data went through the same decontamination check as Gen4; see Disclosures.

3. Training. The base model was trained on the first round. The result was then trained further on the second round, and the two were averaged 50/50. A single temperature was fitted and folded into the weights. It changes the size of the probabilities, but not which option ranks first.

4. Selection. We compared 10 candidates (checkpoints of both rounds and two averages), with Decision Index 0.3 public scores between 68.09 and 69.60. Every candidate had to pass the same checks as Gen4, against KnowLine-4B-Gen4 and against the first-round model: no significant drop on our held-out sets, Open-Jev 1.1 OOD or JevBench-hard; an injection rate of at most 4%; and no regression beyond set margins on new-domain tasks, sibling skills, format consistency, unseen injection templates or calibration (measured after applying the temperature). Gen1 has the highest public score among the candidates that passed: 69.60 before the temperature was folded in, and 69.49 in the fresh run of the released weights (the temperature does not change the ranking, so the difference is run-to-run noise).

5. What the second round did. On the public suite, the second round alone did not add points: its checkpoints scored 68.84-69.14, against 69.40 for the first-round model, and the average scores 69.49-69.60. It did help where the public suite does not look: new-domain tasks 55.0 → 56.4, sibling skills 65.6 → 66.4, Open-Jev 1.1 OOD 89.7 → 90.4. The cost is a small drop on our Simplified Chinese held-out set (−0.8, p = 0.051).

Where the gain over Gen4 comes from

Gen1 and Gen4 were trained on largely the same kinds of tasks. For 33 of the 37 benchmarks, the training data holds items of the same task or from the same source; only MMLU-Pro, BBH, GPQA Diamond and HLE have none. Most of the 4.59 points therefore come from the larger base model rather than from new benchmark data:

• About 3.9 points come from the 33 benchmarks with training data, and about 0.7 from the four without. Per benchmark, the four without training data gained more on average.
• Largest gains, skill Gen4 → Gen1: iSarcasmEval 40.3 → 64.2, CRUXEval 43.8 → 63.3, MMLU-Pro 51.6 → 66.6 (no training data), MMLU 69.9 → 82.2, BBH 53.1 → 64.9 (no training data), ANLI 51.7 → 62.4 and New Yorker caption matching 64.5 → 73.0.
• The largest drops are small: POP909-CL (85.0 → 83.4) and Habermas Machine (48.1 → 46.8).

Comparison

Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.

Board rows as of the board's latest update (2026-10-13).

Public-suite scores by area (DI 0.3):

Releases

The model is trained in a self-evolving loop, and new versions will follow.

Evaluation

All results are self-run and have not been verified by a third party.

Held-out and Chinese evaluations

In a paired sign test against Gen4, the gains on the Simplified and Traditional Chinese in-house sets and on Open-Jev 1.1 OOD are significant (p < 0.001); the differences on the English in-house set and on JevBench-hard are not.

Web operation (Mind2Web official test splits, evaluation only)

• Slightly above Gen4 on all three splits; no request failed.
• So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some degree.
• The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web leaderboard.

Images (zero-shot)

• No image training: Gen1 keeps the base model's vision tower unchanged and saw no images in training. knowline_server.py accepts OpenAI-style image / image_url content parts in a chat state and passes them to SGLang; each question is still one prefill.
• How we measured it: the same reconstruction of the Decision Index 0.3.1 vision public suite as for Gen4: 11 benchmarks, the board's row counts where the sources allow, images capped at 1.6 MP, chance-corrected skill and the board's weights, run through the release server: 12,192 requests, 0 failed.

• Mixed against the stock model: 3.4 points higher overall. The gains are on charts, infographics, web screenshots, exam pages and RealWorldQA; BLINK (multi-image perception, −6.6), Hateful Memes, Winoground, KIE and CV-Bench are slightly lower.
• Indicative only: these are self-run numbers on our reconstruction, not on the board's rows. Our CharXiv distractors lean towards numeric questions, and our Mind2Web marks and KIE questions are our own construction.

Game harness (KOF '98)

• Setup: single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for the 4B line.
• Result: 17-0-1 against Jev 1.13, 18-0-0 against StartLux-Decision-35B-A3B and 12-4-2 against KnowLine-4B-Gen4; 47-4-3 overall, a score of 0.898 [0.79, 0.95].
• Play style: it chooses moves by distance: at close range mostly the 623C anti-air uppercut (45%) and heavy punch (41%), special_2 at mid range (68%), and mostly waiting at long range (64%). Overall, it uses special_2 36% of the time, the uppercut 16%, heavy punch 14%, special_3 13% and waiting 13%, a more varied mix than Gen4's.
• Caveats:
  • 18 games per pair give wide intervals.
  • 27 of the 54 games ended at time-out, so those were decided on remaining health. Gen1 won 25 games by K.O. (13 against StartLux, 9 against Gen4, 3 against Jev); its 2 K.O. losses were both against Gen4. Against Jev, 14 of the 17 wins came at time-out.
  • This is a measured result in this harness, not general fighting-game skill.

Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are used.

• Temperature in the weights: the final norm weight includes a readout temperature T = 1.20, so every logit is divided by 1.20 before the softmax over the option labels. The top-ranked option does not change, so neither do the Decision Index scores; the probabilities are just less extreme. T was fitted on a set of our own evaluation items whose distribution is covered by the training data, not on the held-out sets or on Decision Index rows. On our held-out sets, ECE with the temperature in the weights is 0.025 (Gen4: 0.043).
• Why ranges: the board does not publish which 32 benchmarks it uses. We checked our computation against three board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the plausible benchmark sets.
• Brier score: lower than every model on the board; the lowest there is 0.284 (Torchcast Decision 27B). Part of this comes from higher accuracy, not only from calibration.
• Still slightly overconfident: mean confidence is about 5 points above accuracy. Most of the gap comes from HLE, hard reasoning (MuSR, CRUXEval, GPQA Diamond), cfcolor and New Yorker caption matching.
• Recommendation: the temperature was fitted on our data. If your use depends on probability thresholds, check calibration on your own data.

Disclosures

• Decision Index format training data: in the first round, about 21% of the rows were built in the request format of Decision Index benchmarks (some with reworded instructions), and another 4% are copies of the same items rendered in other formats; in the second round, about 8% and 9%. These include train splits of public datasets rewritten in that format (for example GSM8K, WinoGrande XL, ACOS and Amazon ESCI) and synthetic items written in the same format.
• Decontamination:
  • All training data, of both rounds, was checked with the same tool as Gen4 against every Decision Index 0.3 row, all of our evaluation sets and the other public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM).
  • Rows that nearly duplicate an evaluation item, repeat its question or claim, or contain its whole state were removed: 7,833 of 1,500,018 rows in the first round and 2,445 of 985,519 in the second. Rows that only share a source passage with an evaluation item, such as the same Wikipedia paragraph or tool description, were kept (31,249 and 5,702 rows).
  • BPoMP and cfcolor training items come from the same source datasets as these benchmarks (OEDILF limericks; the O'Donovan et al. colour ratings), with every benchmark item excluded, as in Gen4.
• Selection on evaluations: of the 10 candidates, we picked the one with the highest Decision Index 0.3 public score among those that passed all of our release checks.
• Game data: labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation states.
• No model outputs as labels: no output of Jev or any other decision model was used as a training label.
• Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but not all of them were checked by a human.

Limitations

• Size and speed: it needs much more GPU memory than KnowLine-4B (at least 48 GB with FP8 at load), and each request takes longer: 51 ms median, one request at a time on one H20, against about 31 ms for KnowLine-4B (see INFERENCE.md).
• Knowledge-heavy reasoning: better than the 4B line but still the weakest area (52.5, DI 0.3); HLE remains at chance level.
• Math: answers are given without step-by-step reasoning. The rebuilt 0.3 GSM8K scores 69.3; part of this comes from training on the GSM8K train split rewritten in the same format.
• Prompt injection: on our injection set, planted instructions changed no answers, and about 1% with templates not seen in training. Other attacks may still work, so keep untrusted text clearly delimited.
• Images: there was no image training. The image scores above are zero-shot and self-run on our own reconstruction of the suite, so check them on your own data before relying on them.
• Private tests: part of the public-suite score comes from benchmarks whose tasks are in our training data, so the 0.3 private tests may show a lower score.

Citation

@misc{knowline35ba3bgen1,
  title  = {KnowLine-35B-A3B-Gen1: a 35B-A3B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-35B-A3B-Gen1}
}

License

• Weights: Apache-2.0, the same as the base model.
• Code: knowline_server.py is MIT (see the file header).