Prumo

A decision engine with zero learned parameters. It computes what can be computed and abstains, by construction, from everything else.

Paper: Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters (arXiv:2610.30970)

Model page: huggingface.co/tardellirs/prumo · Code: github.com/tardellirs/prumo

Prumo is Portuguese for plumb line: a tool that is exact without learning anything.

Prumo reads a state and a set of typed questions, the Decision Index / systemone wire format, and returns a probability for every option. It contains no model, no weights, no embeddings and no statistics fitted to a corpus. It is a fixed pipeline of generic capabilities, each of which recognises a structure in the request and solves it exactly:

• code to execute;
• a chess position;
• a catalogue of tools;
• candidates to rank;
• a logical expression;
• a date;
• an arithmetic problem.

When nothing applies, it says so with a uniform distribution.

Size is part of the result. Prumo sits at the zero-parameter end of the size/quality trade-off and puts a floor under it: this much of the public Decision Index is reachable without learning anything.

Highlights

• Full score 12.21 on the Decision Index 0.4 (maintainer-run private tests: same skills 15.78, new domains 5.60), above Crivo-17M (11.11), a learned 17M-parameter model.
• Public index 20.14 on the Decision Index 0.3 (shown on the board, not counted in 0.4), from a complete run of the official kit (140,178 requests). For reference, the other rule-based entrant scores 1.55, and learned models of 17M and 68M parameters score 7.44 and 13.01.
• Exact where it applies:
  • executes the code in CRUXEval (skill 0.997);
  • validates tool calls against their schemas (When2Call 0.70, BFCL 0.67);
  • searches chess positions;
  • solves algorithmic BIG-Bench Hard tasks.
• Honest elsewhere. Knowledge, commonsense and NLI benchmarks stay at chance by design. The engine never guesses beyond what it can compute.
• Fast and tiny:
  • median 0.2 ms per request on one CPU thread, mean 5.1 ms;
  • no GPU, no download beyond ~2.7k lines of Python and one pure-Python dependency.
• Deterministic:
  • every budget is counted in operations, not seconds;
  • the macOS and Linux runs give bit-identical answers.

What it is

How it decides

Every question goes through the same capabilities, in the same order, for every request:

code → chess → tools → retrieval → grounding → consensus → labels → music → logic → puzzles → dates → arith

Each capability either claims a question and returns probabilities, or passes it on. A question that no capability claims gets the uniform distribution, with a deterministic hash pick for the reported choice, so its expected score is exactly chance. The engine never looks at benchmark names, item ids or answer tables, and it uses one response format for every request.

Results

Decision Index 0.3 public suite, run once with this exact code through the official kit (pipeline --edition 0.3; complete: true, all 140,178 requests answered). In edition 0.4 the board shows this public index, but the Full score comes only from the maintainers' private tests (below).

Public index 20.14 (raw 38.96). Skill per area, chance-corrected:

The table below has every benchmark with skill above 0.03. Skill is chance-corrected: 0 is chance and 1 is perfect.

Everything else (MMLU, MMLU-Pro, GPQA, ARC, HellaSwag, WinoGrande, ANLI, NLI4CT, VAST, CLadder, SATA-Bench, HLE, …) is at chance, as intended.

Public index of other submissions, from the board, for scale:

Decision Index 0.4 — Full score (board)

Scored by the maintainers on the board in the Decision Index 0.4 batch of 2026-10-20. In edition 0.4 the Full score is 0.60 × private tests of the same skills + 0.40 × private tasks from new domains, both mapped onto the public scale using untuned stock models. The public index is still run and shown, but it no longer counts toward the Full score, and the new-domain tasks were revised. Edition 0.3.2 (0.20 public + 0.50 same skills + 0.30 new domains) stays available as an archive. Models within 0.9 points of the first model of a group share a rank group.

Rank group 46 is shared with Intern-Decision-0.8B (Full 12.67), Tev1-0.8B-experimental (Full 12.63) and Jev-Style 0.8B v3 (Full 12.07).

With zero learned parameters, Prumo scores above Crivo-17M (11.11, a 17M-parameter learned encoder). The board classes Prumo as an inference technique and, since it has no base model, lists no parameter count for it, so it does not appear on the board's size chart.

Compared with small and rule-based entries (board, Decision Index 0.4):

On the private tests the exact capabilities transfer only in part: the same-skill score (15.78) is below the public index (20.14), and on the private tasks from new domains Prumo scores 5.60 (Zev, the other rule-based entry, 1.89).

Latency

One CPU thread, per request, over the whole public suite (Apple M2):

The slowest paths are the chess search (median 118 ms, bounded by nodes) and the code sandbox (median 41 ms per start). The board's admission limit is 1,000 ms (median, mean or p80).

Determinism

Every budget is counted in operations, never in time:

• chess: 7,000 nodes;
• code: 10⁶ executed lines and calls;
• arithmetic: 60k operations.

The answers therefore do not depend on the machine. Two checks:

• Same machine, different processes: 1,100 requests re-run in a fresh process with a different PYTHONHASHSEED are bit-identical.
• Across operating systems: 3,370 requests (17,258 questions; 80 from each of the suite's 43 datasets) were run on Linux. That host was Ubuntu 24.04, x86-64, CPython 3.12.3, with no OS sandbox available. The answers are identical to the macOS run (Apple M2, CPython 3.12.13, sandbox-exec): agreement 1.0, maximum probability difference 0.0.

Usage

The Decision Index kit is not on PyPI; install it from its repository.

git clone https://github.com/apolinario/decision-index && pip install -e decision-index
git clone https://huggingface.co/tardellirs/prumo && cd prumo      # or github.com/tardellirs/prumo
pip install -r requirements.txt                                     # chess==1.11.2

# the public suite, exactly as submitted
python -m decision_index pipeline --engine decision_index_engine:DecisionEngine --edition 0.3 \
    --suite-dir <decision-index>/suite-0.3 --out runs/prumo

python -m unittest discover -s tests -v     # sandbox, determinism and latency checks (~5 s)

In Python:

from decision_index_engine import DecisionEngine

engine = DecisionEngine()

state = {"code": "def f(xs):\n    return [x * 2 for x in xs if x % 2]", "call": "f([1, 2, 3, 4, 5])"}
questions = {"output": {"type": "choice", "instructions": "What does the call return?",
                        "criteria": {"A": "[2, 6, 10]", "B": "[2, 4, 6, 8, 10]", "C": "[1, 3, 5]", "D": "[]"}}}
response, trace = engine(state, questions)
response["answers"]["output"]["choice"]   # "A"
trace                                     # {"solved_by": {"output": "code", "code_ms": 52.9}}

A question that no capability recognises comes back with uniform probabilities, and trace["solved_by"] stays empty for it. That is the signal to send the question to a learned model. Prumo works well as a zero-cost first stage in front of one.

To run without executing any code from the request, leave out the code capability:

--option solvers=chess,tools,retrieval,grounding,consensus,labels,music,logic,puzzles,dates,arith

Safety: executing code from requests

Code from a request never runs in the engine's own process. It runs in a child python -I -S -B, in a fresh empty temporary directory, with a minimal environment. The protections come in layers:

1. OS sandbox, when the host offers one:

   • macOS: sandbox-exec (no network, no writes outside the temp dir);
   • Linux: bwrap (all namespaces unshared, read-only root), or else unshare -rn.

   On hosts where unprivileged namespaces are blocked, the engine detects it and relies on layers 2–4. Ubuntu 24.04 with default AppArmor is one such host.

2. Resource limits: 1 s of CPU, 2 GiB of address space, 1 MiB per file, no new processes.

3. A CPython audit hook, installed before the code runs. It refuses:

   • sockets and network modules;
   • subprocess, fork, exec and spawn;
   • ctypes;
   • writes outside the temp dir;
   • reads outside the Python installation and the temp dir.

4. A deterministic budget of 10⁶ executed lines and calls, plus a 1 s wall-clock safety net.

A run that hits a limit counts as "cannot tell", and the question falls through. The tests in tests/test_engine.py check every layer on each host: no network, no writes, no reads of user files, no processes, budget and timeout. They pass on macOS and on Linux.

Transparency

• No learned values. Every number that shapes a decision is one of the 527 hand-set parameters in model.safetensors. The engine loads them at start-up (prumo_rules/params.py, standard library only, F64, so the values are bit-exact). tools/make_constants.py is their human-readable source, with a comment on each. The file holds:

  • 454 chess values: piece values plus seven textbook piece-square tables;
  • search constants;
  • the code, arithmetic and sandbox budgets;
  • BM25 k1 = 1.2, b = 0.75;
  • every threshold, weight, temperature and confidence mass.

  The file is named model.safetensors so that the Hub counts it, but it is not a trained model: it is the engine's configuration, kept where the size can be counted. Word lists (stop words, phrase lists, month and number words) and structural constants (board geometry, the calendar, the 12 pitch classes, text-length limits) stay in code. The thresholds that could not be fixed a priori were chosen on training/dev splits of the source datasets and on our own panels, never on the public test.

• Removed before release. An earlier prototype scored 24.81 on the public index, but it had parsers keyed to the fixed wording of particular benchmarks, and a few rules inspired by reading errors on the public test. All of them were removed:

  • the CLadder template parser;
  • the synthetic-phishing URL heuristics;
  • BIG-Bench Hard phrase anchors and word tables;
  • the dialogue boosts of API-Bank and When2Call.

  The tool rules were rebuilt from generic principles (schema validation, relatedness, recency) and validated on training data and on generated paraphrases. The public suite was then run once, for measurement only.

• Remaining caveat. A few constants of the prototype were first tried on public subsets before they were frozen: the chess evaluation, the chord templates, the arithmetic reachability budget and the consensus scoring. They are generic and kept, and they are declared here.

Limitations

• No world knowledge, commonsense or language understanding. MMLU-style, NLI, sentiment and stance tasks are at chance by design.
• Recognition is structural. A request whose structure the parsers do not recognise falls back to chance. Examples: code without a def, tools described only in prose, dates without an anchor. Prumo does not degrade gracefully toward partial understanding the way a learned model does.
• The lexical capabilities are heuristics (labels, grounding, retrieval, consensus). They help, but they are far from neural quality.
• Executing dataset code is a risk you should evaluate. Read the safety section, or disable code.

License

Apache-2.0 for this repository. The chess dependency (python-chess) is GPL-3.0 and is installed separately, not bundled.

Citation

@misc{stekel2026jointperoption,
  title         = {Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters},
  author        = {Tardelli Ronan Coelho Stekel},
  year          = {2026},
  eprint        = {2610.30970},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.30970}
}

@software{prumo2026,
  title  = {Prumo: a zero-parameter rule-based decision engine},
  author = {Tardelli Stekel},
  year   = {2026},
  url    = {https://huggingface.co/tardellirs/prumo}
}