Whittle-Reflex-25B-A3B (early preview, v4)

⚠️ Early preview

This is an early preview release of a research model, published so people can try it and send feedback.

• The scores below come from a full run of the S1MB english-v1 suite (137 benchmarks, the suite's own evaluator) and my own rebuild of the Decision Index 0.1 suite (a 40-question slice per benchmark). Neither is an official board listing yet.
• The model, the files and these numbers may change in later versions. Pin a revision if you depend on it.
• It was trained only to pick among options. It is not a chat model.
• v4 (10 Oct 2026) replaces v3: trained further on the S1MB author's own training release (train splits of the sources S1MB tests, in S1MB's question format), on top of v3. Read the overlap notes before comparing its S1MB score with other models. v3 stays downloadable at the v3 tag of this repo, v2 at v2, v1 at v1.

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

A System 1 decision model, not a chat model. Give it a state, a question and a set of options; it reads them once and returns a calibrated probability for every option, in a single forward pass. No generation, no thinking tokens.

It runs on official llama.cpp (no patches): the GGUFs carry the decision metadata, so llama-server serves the POST /v1/systemone endpoint out of the box (decision type pplx-decider). See How to run.

• Size: 25 B parameters, ~3 B active per token (mixture-of-experts, the Whittle 25B body).
• GGUFs (recommended): logic65/Whittle-Reflex-25B-A3B-GGUF.
• This repo: the bf16 root weights (safetensors), for research and fine-tuning.

How to run

The easy path is a GGUF with llama.cpp's built-in decision endpoint.

1. Get llama.cpp build b11455 or newer (tested on b11455). Prebuilt binaries are on the llama.cpp releases page, or build it:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON          # drop -DGGML_CUDA=ON for a CPU-only build
cmake --build build --config Release -j --target llama-server

2. Download a GGUF (Q4_K_M is the smallest one I recommend):

hf download logic65/Whittle-Reflex-25B-A3B-GGUF Whittle-Reflex-25B-A3B-Q4_K_M.gguf --local-dir .

3. Start the server:

./build/bin/llama-server -m Whittle-Reflex-25B-A3B-Q4_K_M.gguf -ngl 99 -fa on -c 32768 --port 8080

• The log line decision model type: pplx-decider shows the decision metadata was read.
• Not enough GPU memory: add --n-cpu-moe 20 (keeps the expert weights of the first 20 layers in system RAM; raise it until the model fits).
• No GPU: leave out -ngl 99. It works on CPU, just slower.
• -c is the context for one decision (state + question + options). 32768 covers long states; use less to save memory.

4. Ask a question:

curl -s http://localhost:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"note": "geography quiz"},
  "questions": {
    "pick": {"type": "choice", "instructions": "Which city is the capital of Australia?",
             "criteria": {"sydney": "Sydney", "canberra": "Canberra", "melbourne": "Melbourne"}},
    "noul": {"type": "noul", "instructions": "Is the Pacific the largest ocean?"}}}'

The answers from the Q8_0 file:

{"answers": {
  "pick": {"type": "choice", "choice": "canberra",
           "probabilities": {"sydney": 0.0058, "canberra": 0.9873, "melbourne": 0.0069}, "confidence": 0.981},
  "noul": {"type": "noul", "noul": 0.9867}}}

The same from Python:

import requests

r = requests.post("http://localhost:8080/v1/systemone", json={
    "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
    "questions": {
        "route": {"type": "choice", "instructions": "Which team should handle this?",
                  "criteria": {"billing": None, "shipping": None, "technical": None}},
        "urgency": {"type": "score", "instructions": "How urgent is this?",
                    "criteria": ["can wait", "this week", "today", "right now"]},
    },
})
for qid, a in r.json()["answers"].items():
    print(qid, a)

Question types:

• state can be a string, an object or an array; anything that is not a string is given to the model as JSON.
• Each question in a request is answered independently, so you can send several at once.
• Full endpoint reference: the /v1/systemone section of the llama-server README.

transformers (root weights)

from whittle_load import load_whittle   # in this repo; sizes the (empty) n-gram memory from the checkpoint
model = load_whittle("logic65/Whittle-Reflex-25B-A3B")

The root weights load in stock transformers (tested with 5.18) with whittle_load.py. They are meant for research and further training. For decisions, serve a GGUF as above. If you want the readout in your own code: the prompt is systemone.jinja (in this repo), options get the single-token labels A, B, C, ... AA..JT, and the answer is the softmax over those labels' next-token logits.

Results

S1MB english-v1 (full suite, 137 benchmarks)

S1MB is a public System 1 benchmark: 137 typed decision benchmarks (yes/no, choice and score questions), each scored 0 to 100 after subtracting a trivial baseline, equal weight per benchmark. The Reflex rows are my own runs with the suite's evaluator (official llama.cpp b11455, the Q8_0 file, 137/137 complete, zero failed decisions); the other rows are the top entries of the suite's public results (hotchpotch/s1mb-result, commit b9c4c60d, 102 complete entries), ranked in the board's default order (mean rank over all 137 benchmarks).

• Placed among those 102 entries, v4 would rank 2nd on the mean of the three columns (66.4; Darwin-27B-ZTC-v2 66.5) and 4th in the board's default order. Its score column (63.1) is the highest of any complete entry. It is not listed on the board yet; a results submission is pending.
• v4 vs v3: the 98 benchmarks outside Open-Jev went from 47.3 / 54.0 / 35.8 to 54.8 / 65.4 / 47.6 (73 up, 17 down); the 39 Open-Jev benchmarks held (88.6 / 87.3 / 93.9 to 89.4 / 86.8 / 94.1).
• Largest gains: temporal NLI (choice) +53, BoardgameQA +52, poem sentiment +51, NanoHotpotQA relevance (yes/no) +51, TRACIE +42.
• Largest drops: ETHICS (choice) -17, BBQ -12, NanoNFCorpus relevance (yes/no) -11, one Open-Jev invoice workflow set -10, SciCite -7.

Overlap, please read before comparing. v4 trained on the train splits of hotchpotch/bekko-system-one-dataset-v0, the S1MB author's training release, built with the same converter as the S1MB test set. So most S1MB benchmarks are in-distribution for v4:

• Benchmarks whose source dataset had training rows rose from 47.3 to 59.8 on average (44 up, 6 down). Benchmarks without training rows rose less, from 50.3 to 56.8 (29 up, 11 down), and part of that is the relevance tests sharing one question template with training rows.
• Training rows whose input matched an S1MB test input were removed (exact input hash, and the state compared after whitespace normalisation). Matching is exact only: some training rows share question templates and scenario names with test rows (the typed decision scenarios, NLI hypothesis templates).
• The 39 Open-Jev benchmarks are drawn from Open-Jev's test split; v3 trained on its train split (no shared rows, per the dataset's published audit). AG News, MASSIVE and SNLI train splits were in v2's data.

Decision Index (unofficial slice)

Measured on 1,173 requests: the first 40 per benchmark of the Decision Index 0.1 suite, 28 benchmarks. This is my own rebuild of the suite, not an official board run:

• HLE is missing (gated).
• ToolRet, BRIGHT and ACOS are not in the slice.

Mean chance-corrected skill (0 = chance, 1 = perfect):

• v4 vs v3: +0.028 on stock serving and +0.041 with two orders, better on 14 benchmarks and worse on 8 either way (sign test p = 0.29, not significant on its own).
• Biggest gains: ContractNLI (0.51 to 0.74), Amazon ESCI (0.28 to 0.50), CLadder (0.25 to 0.40), GSM8K (0.20 to 0.30; 0.40 with two orders), NLI4CT (0.30 to 0.39; 0.53 with two orders), ForecastBench (-0.04 to 0.09).
• Slipped: MuSR (0.24 to 0.09), API-Bank (0.56 to 0.46), CLINC150 (0.73 to 0.65), GPQA Diamond (0.10 to 0.03).
• Earlier steps: v2 beat v1 on 16 benchmarks and lost 5 with two orders (sign test p = 0.027); v3 was flat against v2 on this slice.
• 40 questions per benchmark is a small sample: single-benchmark numbers are noisy, the mean is the figure to trust.

A full Decision Index 0.3 run of v2 (the board's own kit, stock llama.cpp, Q8_0) was started and stopped early; what it scored:

Speed

One decision per request, no generation. Measured on v1 (same architecture and file sizes as v2 to v4). On two Tesla T4s (Kaggle), Q8_0, stock llama-server:

• median 345 ms per request, p90 472 ms, over 17,913 requests;
• long-state benchmarks take longer (Home appliance simulator: 21 s median, because its prompts are very long).

Per GGUF quant (same slice, via llama.cpp)

Scores come from llama.cpp's own hidden states for each file (read with a llama.cpp build that has a 2-line fix for --embeddings on this architecture; serving uses official llama.cpp). Q8_0 is the v4 LoRA merged into the Whittle 25B Q8_0 (the file the full-suite run uses); Q6_K and Q4_K_M are quantized from the bf16 root weights in this repo with official llama.cpp. The last column is the share of the 64 check prompts where the file picks the same answer as the trained model in PyTorch, and the mean largest probability gap.

Held-out questions from the training tasks

1,013 held-out questions (449 from the new training tasks, plus held-out Open-Jev and v2 questions):

• accuracy: 80.1% with the v3 weights, 84.5% after v4 (new tasks 72.0% to 81.5%, Open-Jev 92.8% to 91.9%, v2's tasks 79.4% to 81.6%);
• calibration error (ECE): 0.029 to 0.014.

How it was made

1. Body: the Whittle 25B, which is the Whittle-Qwen-3.8-35B-A3B body with its n-gram memory switched off.
2. Prompt: every question becomes one fixed prompt (STATE / QUESTION / OPTIONS), with single-token labels A, B, C, ... AA..JT (up to 255 options). The answer is the softmax over the presented labels' next-token logits. This is the same readout llama.cpp's pplx-decider type uses.
3. Distillation (v2): LoRA rank 32 on 250 projection matrices (attention, linear attention, shared expert), trained to match the option probabilities of Qwen3.8-27B read the same way, plus a small gold-label term.
   • Data: 34,853 questions from train splits only: v1's 16,000 (classification and NLI sets) plus 18,853 reasoning-style ones (ARC, HellaSwag, WinoGrande, OpenBookQA, QASC, WANLI, MedMCQA, AQuA-RAT).
   • Options are shuffled, so the right answer sits in a random position. 27 items whose wording matched a Decision Index question were removed. No Decision Index test items were used.
4. Control tasks (v3): the v2 LoRA trained further on 20,960 questions from the train splits of ZefanCai/Open-Jev (CC0; 22 synthetic control tasks), with the dataset's own reference probabilities as targets, plus all v2 questions replayed. Yes/no questions are rendered the way llama.cpp's pplx-decider serving renders them (false first, with the standard option descriptions).
5. S1MB-format training (v4): the v3 LoRA trained further for one epoch (1,755 steps, 116 minutes on one RTX PRO 6000) on 53,249 questions (18.2 M tokens):
   • 40,236 decisions from the train splits of 114 subsets of hotchpotch/bekko-system-one-dataset-v0 (revision 9c673a09), with the dataset's reference probabilities as targets. They are rendered exactly as S1MB sends its questions to llama.cpp (checked against the suite's own request builder on all 29,469 S1MB test decisions). At most 3 decisions per case, yes/no balanced 50/50, and more rows for the benchmarks where v3 was weakest.
   • Subsets were left out when their source licence is non-commercial, GPL/LGPL, unknown, unconfirmed or access-restricted (19 subsets), and the Open-Jev copies were left out (v3 already trained on Open-Jev's own release). Ranking decisions and decisions with documents were skipped.
   • Every training decision whose input matched an S1MB test input was removed (see the overlap notes above).
   • Replay: 6,000 Open-Jev and 6,000 v2 questions, so the earlier skills stay.
6. Merge: the LoRA is merged into the weights. The Q8_0 GGUF is the LoRA merged straight into the Whittle 25B Q8_0 (the exact file every benchmark above used); the root weights, Q6_K and Q4_K_M are built from the bf16 merge with official llama.cpp tools, with the decision metadata and route-2 template baked in.

v1 (7 Oct) used Qwen3.6-35B-A3B as the teacher and only the 16,000 classification questions; v2 adds the reasoning set and the stronger teacher; v3 adds the Open-Jev control tasks; v4 adds the S1MB-format training release.

Limits

• Early preview. Expect changes; the Decision Index full-suite score is not in yet.
• Not a chat model. It was trained only to pick among options. Chat behaviour is reported below for transparency, not as a use case.
• Single option order on stock llama.cpp. Stock pplx-decider serving reads one option order (0.423 on the slice, Q8_0). Averaging two orders (0.433) needs your own wrapper; llama.cpp averages orders only for lev-type models.
• Yes/no questions are read as true/false, not on a rating scale.
• Text only. There is no vision projector for this model.
• English only. All training questions are English.
• Training text sources. The question texts come from the public datasets above; the bekko subsets keep their source licences (listed in that dataset's sources.json). Check their terms for your use.
• Task overlap (Decision Index). Several training sets share a task type with Decision Index benchmarks (for example ARC, HellaSwag and WinoGrande train splits). Only train splits were used, and any question matching a suite item was removed.
• Benchmark overlap (S1MB). v4 trained on train splits of most S1MB source datasets, in S1MB's own format; the 39 Open-Jev benchmarks share a source with v3's training data; AG News, MASSIVE and SNLI train splits were in v2's data. Its S1MB score measures these task families in distribution, not unseen-task generalisation.
• Weak spots that got worse in v4: ETHICS and BBQ (bias) choice questions, MuSR, API-Bank.

Chat behaviour (for transparency, measured on v1)

Same review battery as the other Whittle models, base 25B vs Reflex v1, same harness and machine (not rerun for v2, v3 or v4):

Same Q8_0 files, same llama.cpp build, same sampler (temperature 0.7, top-p 0.8, top-k 20, repeat penalty 1.05), thinking on, one RTX PRO 6000. The heal left general behaviour roughly where it was (better on the agent and web probes, worse on exact JSON formatting); it is still published as a decision model, because that is what it was trained and measured for. MATH-60 was not run.