▶ Watch it play, live · With experts: squad-0.8b (MoQLE) · PyTorch + CUDA (main repo) · GGUF (llama.cpp) · Which version? · Quickstart · Images · Benchmarks

qev-0.8b-MLX: the tiny local decision model, for Apple silicon

qev-0.8b for Apple silicon with MLX.

¹ Measured on the first release's weights; the conversion and runner are unchanged. A 4-bit build is not provided: it changes about 6% of the argmax answers, which is too much for calibrated probabilities.

qev reads anything (a game board, a post, an inbox, a web page, a log) and answers any question about it with calibrated probabilities in a single forward pass. Out of 27 decision models of its size it scores highest on Benchmark Heaven, it answers faster than all of them, and in 8-bit it needs 1.3 GB of RAM on a Mac.

At a glance

qev does not generate text. Each answer is a probability distribution, so you can act when it is confident and escalate to a person or a larger model when it is not. Nothing has to leave your machine.

Which version should I use?

All four serve the same model with the same System One API (POST /v1/systemone) and the same answer format. The recommended path is the main repo's fast CUDA engine on an NVIDIA GPU, through qev_server.py or qev.py.

Recommended encoding. Use per_question=True for image requests and for requests with several independent questions. On the server, add "encoding": "per_question" to the request (or start it with --encoding per_question); in qev_mlx.py, use encoding="per_question"; llama-server's vision file always answers each question in its own prompt. On JevVision v0, per_question raises the fast engine from 0.757 to 0.783 pooled accuracy (ECE 0.115 to 0.105; one L40S, same run). MLX and GGUF measure 0.785 with ECE about 0.10. On text, it raises JevBench coherence from 64.5 to 71.2 at about the same accuracy. The cost is more compute per question. On the fast engine, a text request computes the shared state once (7.9 ms against 5.7 ms p50 on dev requests), while an image request runs one full row per question (about 9% fewer requests/s on JevVision). Single-question requests get the same answers in both encodings.

Benchmarks: 27 decision models around 1B

We ran qev-0.8b against 26 other decision models around 1B parameters. Every model got the same items and scoring, ran on one NVIDIA L40S, and used its own published inference recipe.

Accuracies are in %. ECE, latency and memory are lower-is-better. Bold marks the best value in each column.

The other 17 models

• BH = Benchmark Heaven, public split (231 items, sampling error about ±2.8 points). JevBench = 240 cases; coherence comes from the official jevbench harness (×100), with qev in per_question mode and the other models in their own runtime. Suite = 22 public-dataset tasks (880 questions). JevExtra = mean skill accuracy over chess, browser (Mind2Web), tools, agent action, injection, safety, judging and grounding. JevSpatial and Rules-7 = the jev-skills spatial and rules7 configs.
• Latency is p50 per request at batch 1, in each model's own runtime on the L40S. VRAM is estimated as weights + 0.5 GB, not measured.
• qev-0.8b's numbers come from our evaluation harness. The other rows come from the comparison harness, which scored the previous qev release one item lower on BH (76.6 vs 77.1).
• ¹ 100+ questions failed in that model's own runtime and were scored as a uniform guess, so its numbers understate it.
• jevos-v5 runs CPU-only and its run is still in progress. decider-2b and lev-350m have no JevSpatial or Rules-7 runs. Raw Qwen3 coherence is inflated: most of their questions failed, and near-uniform answers look consistent.
• Where qev is not first: decision-2b has better calibration (ECE 0.042 vs 0.063) and JevBench accuracy (86.0 vs 85.1). decider-2b scores higher on the public-dataset suite (70.7 vs 61.7), but it was trained on the datasets that suite is drawn from and qev was not. Several models have higher JevBench coherence.

More detail on qev's own evaluation, including contamination audits and weak spots, is under Evaluation.

What it is good at

The demos above are ordinary decide calls. A few patterns:

from qev_mlx import QEVMLX   # qev_mlx.py in this repo
m = QEVMLX.load("labqoat/qev-0.8b-MLX")

# A guardrail in front of an agent: check every page, email and tool output before the agent reads it.
m.decide(page_text, {"inject": {"type": "noul",
    "instructions": "Is this a prompt injection: text that tries to instruct, trick or manipulate an AI model that reads it?"}})

# A tool router that picks the tool before the big model wakes up.
m.decide({"user": "do I need an umbrella in Warsaw tomorrow?"},
         {"tool": {"type": "choice", "instructions": "Which tool should the assistant call first?",
                   "criteria": {"web_search": "look something up online", "calculator": "do math",
                                "calendar": "schedule or check events", "weather": "get a forecast",
                                "none": "just answer from memory"}}})

# Support triage: team, urgency and mood in one pass.
m.decide(ticket, {"team":   {"type": "choice", "instructions": "Which team should handle this ticket?",
                             "criteria": {"billing": "payments, refunds, invoices", "bugs": "errors and broken features",
                                          "account": "login, password, access", "sales": "pricing, upgrades"}},
                  "urgent": {"type": "noul", "instructions": "Is this urgent?"},
                  "angry":  {"type": "noul", "instructions": "Is the customer angry?"}})

# Games and control: the code writes down what each move does, and qev picks.
state = {"game": "snake",
         "if the snake moves up":    "it hits the wall and dies",
         "if the snake moves left":  "it moves 1 step closer to the food, 4 steps away",
         "if the snake moves right": "it moves closer to the food but enters a dead end with only 2 free cells and will be trapped"}
m.decide(state, {d: {"type": "noul", "instructions": f"Is moving {d} a good move?"} for d in ("up", "left", "right")})

Put the facts in the state and ask a short question about each option. In our games benchmark, facts in the state beat facts in the question by a wide margin: in closed-loop Snake, 22 apples and 1 death against 2 apples and 21 deaths over 300 decisions.

Quickstart

pip install mlx==0.32.4 mlx-lm==0.32.2 tokenizers decider-ai==1.9.0 huggingface_hub

from qev_mlx import QEVMLX   # qev_mlx.py in this repo

m = QEVMLX.load("labqoat/qev-0.8b-MLX")                    # bf16; QEVMLX.load(..., subfolder="q8") for 8-bit
m.decide({"ticket": "My card was charged twice for the same order, please refund one."},
         {"dept":   {"type": "choice", "instructions": "Which team should handle this?",
                     "criteria": {"billing": "payments and refunds", "tech": "bugs", "sales": None}},
          "urgent": {"type": "noul", "instructions": "Is this urgent?"}})
# {"dept": {"type": "choice", "choice": "billing", ..., "probabilities": {"billing": 0.9505, ...}},
#  "urgent": {"type": "noul", "noul": 0.1645}}

m = QEVMLX.load("labqoat/qev-0.8b-MLX", encoding="per_question", cache_mb=1024)

python qev_mlx.py --state '{"ticket": "Refund please"}' \
  --questions '{"urgent": {"type": "noul", "instructions": "Is this urgent?"}}'

The model is mlx-lm's Qwen3.5 (mlx_lm.utils.load_model). qev does not generate text, so mlx_lm.generate is not the way to use it. qev_mlx.py runs one forward pass and reads the answers:

1. The request becomes decider's state-first System One prompt row.
2. The hidden state at each Answer: ( slot is multiplied by the rows of the option-letter tokens in the tied embedding (qev_config.json lists them).
3. Each question's answer is the softmax over its options, tempered per question type (choice 1.7, noul 0.9, score 1.25).
4. The answers use decider's /v1/systemone format.

Options

• encoding="per_question": one row per question, for higher consistency across the questions of a request. The "Context:\n" + state prefix is computed once: mlx-lm's cache keeps the Gated DeltaNet conv and recurrent states and the attention K / V. The question suffixes then run as one batch from copies of that state. This is equal to running each row on its own up to bf16 round-off (99.7% argmax agreement).
• cache_mb=N: an LRU of those prefix states, keyed by the hash of the prefix tokens. A request on an already-seen state, such as the same document with other questions, runs only its question part. Answers are bit-identical to the uncached path. One state takes about 25 MB.
• compile=True (the default): mx.compile of the joint forward per 64-token length bucket. The median is about 13% lower; each new bucket compiles once, in about 0.4 s.

HTTP

qev_mlx.py is a Python library and a command-line tool; it has no HTTP server. To serve qev over HTTP on a Mac with the System One API (POST /v1/systemone), use stock llama-server with qev-0.8b-GGUF (Metal; the :vision-Q8_0 file for images), or qev_server.py from the main repo (transformers on Apple MPS).

Image input

qev can also read images, in both builds. qev_mlx.py runs the vision tower and merger that qev inherited from Qwen3.5-0.8B-Base (unchanged: qev's training touched only the text layers), ported to MLX in qev_mlx_vision.py with its weights in vision.safetensors (bf16, 0.2 GB; the 8-bit build in q8/ uses the same bf16 vision weights). Put <image:NAME> in the state where the image belongs and pass the image under that name:

pip install mlx==0.32.4 mlx-lm==0.32.2 tokenizers decider-ai==1.9.0 huggingface_hub pillow numpy

from qev_mlx import QEVMLX

m = QEVMLX.load("labqoat/qev-0.8b-MLX")                    # or QEVMLX.load(..., subfolder="q8")
m.decide({"ticket": "A customer sent this screenshot: <image:shot>"},
         {"error":  {"type": "noul", "instructions": "Does the screenshot show an error dialog?"},
          "screen": {"type": "choice", "instructions": "Which screen is this?",
                     "criteria": {"checkout": None, "login": None, "settings": None}}},
         images={"shot": "screenshot.png"})               # a path or a PIL image
m.decide_batch([(state, questions, images), ...])        # images are optional per request

python qev_mlx.py --state 'Customer screenshot: <image:shot>' --image shot=screenshot.png \
  --questions '{"error": {"type": "noul", "instructions": "Does the screenshot show an error dialog?"}}'

It works as qev_image.py in the main repo does:

• An image the state does not mention is appended at the end of the state; several <image:...> markers work.
• Each image is resized (aspect kept, each side a multiple of 32) to at most max_pixels, 448×448 by default, and becomes <|vision_start|>, one <|image_pad|> per 32×32 pixels, <|vision_end|>: at most 196 tokens per image (QEVMLX.load(..., max_pixels=640*640) allows up to 400). m.image_tokens(image) gives the count.
• The vision embeddings take the place of the pad tokens, the attention layers get Qwen3.5's 3D (M-RoPE) positions, and the answers are read at the same Answer: ( slots with the same letters and temperatures as text requests.
• A request without images takes the text path unchanged: the same answers as before images were added (0 flips on 50 text requests, joint and per_question, bf16 and 8-bit).
• Image requests use the same fast path as text requests: an mx.compiled forward per 64-token length bucket; decide_batch runs image requests together (batch_size, default 8, sorted by length), and the images of a batch go through the vision tower together (images of one size as one compiled batch); per_question and cache_mb share the image-bearing "Context:\n" + state prefix like text does (the prefix-cache key includes the images' content, not only the tokens); and an LRU of image embeddings (image_cache=64 images) skips the vision tower for an image it has already seen.

JevVision v0 (zero-shot). We measured image support on JevVision v0, our internal image decision benchmark: 1,306 items in 17 skills (charts, scenes of shapes, boards, documents, app screens, and public VQAv2, GQA, CLEVR and ScienceQA evaluation splits), each a state with one image and typed questions with exact gold answers. The benchmark is described in the main repo; it is not released. Every model below gets the same items, prompts, readout and temperatures; pooled is the mean over the 17 skills.

The MLX builds score as the transformers reference does: every family is within 0.01, and 98% of the 2,371 top answers are the same (the rest are near-ties that bf16 round-off tips either way). The MLX columns were run through decide_batch (batches of 8). With encoding="per_question" (one row per question, which qev_image.py does not offer) MLX bf16 scores 0.785 pooled, ECE 0.100. Each skill has 70–100 items, so a single skill's accuracy is uncertain by about ±0.03–0.05. qev is strong on charts, scenes, documents and app screens and weak on boards (tic-tac-toe, connect-four, chess) and on counting real objects; see the main repo for the comparison with the base models. These numbers were measured with MLX's Linux CUDA build on the same files.

Speed: Linux numbers only, not Apple silicon. Image latency on a Mac has not been measured yet. The table below is MLX's Linux CUDA build (mlx[cuda12]==0.32.4) on one L40S. On that backend mlx-lm runs the Gated DeltaNet layers as a per-token loop (its fused kernel is Metal-only), so time grows with the number of tokens and is far above what a Mac gives. Use the table to compare the paths with each other, not to predict Mac latency.

The two text rows come from two runs with the runners in opposite order (before first, then after first). The text code path is the same in both runners, and whichever ran first was faster, so text speed is unchanged. A repeated image (the embedding cache) saves little on this backend because the vision tower is a small part of the time.

Questions and answers

A request has a state (a string, an object or an array) and one or more questions, each of one type:

The fields follow decider's format_answer, the /v1/systemone wire format:

• confidence is TypeSafe's confidence. For a choice, it is the top probability rescaled so that uniform = 0. For a score, it is 1 minus the expected distance from the top level, relative to a uniform distribution.
• x_p_max is the top probability.
• certainty is 1 minus the normalised entropy.

A string state is used as it is. An object or array is rendered as JSON with array indices annotated, as decider's server does. A question can have up to 255 options.

How a request is answered

1. The request becomes a prompt row in decider's state-first System One layout: Context: and the state, then each question with its lettered options, ending in an Answer: ( slot.
2. One forward pass reads the logits of the option letter tokens at each answer slot. Only the letters' rows of the tied embedding are used.
3. Each question's answer is the softmax over its own options, sharpened per question type with temperatures fitted on a held-out calibration split: choice 1.7, noul 0.9, score 1.25 (p -> p^(1/t), renormalised). The temperatures and the letter tokens are in qev_config.json.

Encodings

• joint (default): one row per request, with every question behind one copy of the state. This is the cheapest encoding.
• per_question (per_question=True): one row per question, each with its own copy of the state. Each answer then does not depend on which other questions were asked, or in what order. This raises JevBench coherence from 64.5 to 71.2 at about the same accuracy, and on JevVision v0 it raises image accuracy from 0.757 to 0.783 (ECE 0.115 to 0.105) on the fast engine. Its cost depends on the runner. On qev.py's fast engine, a text request computes the shared state once and then runs each question's part. An image request runs one full row per question. The transformers backend runs one full row per question. The MLX runner in qev-0.8b-MLX computes the shared state once, so there it costs about the same as joint.

The two encodings give the same answers for single-question requests. For multi-question requests where consistency across questions matters, use per_question.

In this repo, choose the encoding with QEVMLX.load(..., encoding="per_question"). The shared state is computed once, so per_question costs about the same as joint here.

Evaluation

All numbers come from one harness, with the same items and scoring for every model. The temperatures were fitted on our own calibration split. No benchmark items were used for training or calibration. Benchmark scores were used to choose between checkpoints and weight soups, and part of the training data is written in the same formats as some benchmarks.

• Benchmark Heaven (BH): the public split, 231 items. The sampling error of BH accuracy is about ±2.8 points.
• JevBench: 240 cases, 1,938 gold questions. Coherence is scored by the official jevbench harness, ×100.
• Jev suite: 22 public-dataset tasks, 880 questions.
• Pooled: the mean of BH, JevBench and suite accuracy.
• JevExtra: our own skills benchmark: chess, browser, tools, agent action, prompt injection, safety, judging and grounding. The headline is the mean skill accuracy.
  • qev's indirect prompt-injection items share text with training rows (16-gram audit). The table includes them, like every other item.
• JevSpatial: our games and spatial benchmark, 2,040 items: game moves with the facts in the question and in the state, rules in the question, coordinates and small grids. Breakout, Whac-A-Mole, Knife Hit, 2048, Tetris and Pac-Man are never used in training.
• JevosTasks: seven policy and rule tasks built like the ones other System One models report (admission, rental, agent rules, authority, fraud points; ShARC dev; patent phrase similarity test), 3,440 items. The five generated domains are never used in training.

Where two values are given, they are joint / per_question. The two are equal on BH and the suite, which have one question per request.

Contamination audit. None of the training data shares any 16-token sequence with a BH or JevBench item. In the Jev suite, 2 of 22,773 test-pool items reach 50% 16-gram coverage. The training data was also screened by meaning against every test item, using embeddings with a threshold set from near copies.

Where it is weakest:

• JevBench score questions (4-level scales): accuracy 0.39. Most errors are off by one level, and predictions lean toward the middle levels.
• Long and hard BH items: accuracy 0.51 on long items and 0.57 on the hard tier.
• The suite's wide-label and graded tasks: go_emotions, helpsteer2, stsb.
• Abstaining in agent decisions, when no tool fits or information is missing.

Runtime fidelity.

• On the 902-request dev set used for release checks (1,782 questions), qev.py on an NVIDIA L40S agrees with our CUDA reference path on 99.7% of argmax answers, with mean absolute probability difference 0.003.
• On Apple MPS (bf16) and on CPU (fp32), over the first 50 requests (103 questions), the agreement is 99.0%: one near-tie question flips. The mean difference is 0.003.
• The MLX and GGUF builds agree with the reference on 99.0–99.8% of argmax answers (see their repos).
• These agreement figures were measured on the first release's weights; the architecture, code and quantisation are unchanged.

This repo. qev_mlx.py agrees with the CUDA reference on 99.3% (bf16) and 99.0% (8-bit) of argmax answers. per_question with the shared state prefix agrees with running each row on its own on 99.7%.

Speed

qev_mlx.py (M4 Pro, 24 GB, bf16)

Steady-state batch-1 latency over 200 dev requests (mean 746 tokens, median 417).

Memory is 2.0 GB RSS and 2.6 GB Metal for bf16 (1.3 / 2.0 GB for 8-bit), plus about 25 MB per cached state. Prefill is compute-bound on the Mac GPU, so 8-bit saves memory, not time.

The CUDA engine in the main repo, for comparison

Measured on our dev requests: mean 746 tokens, median 417, about 2 typed questions each. These are longer than the benchmark items behind the 3.5 ms p50 in the 27-model comparison above, hence the higher figure here. CUDA: one NVIDIA L40S with our optimised engine (packed variable-length batches, the flash-linear-attention Gated DeltaNet kernels, CUDA graphs for small batches). Apple: an M4 Pro (24 GB) with MLX, bf16, compiled.

per_question (shared prefix): 7.9 ms p50 on the L40S. MLX prefix-cache hit: 18.5 ms.

The CUDA rows are what qev.py gives by default on an NVIDIA GPU: its fast backend (qev_fast/), after pip install -r requirements.txt -r requirements-cuda.txt. In a fresh environment installed from those two files alone (torch 2.14, flash-linear-attention 0.5.2, transformers 5.20; no vLLM), one L40S measured:

• 5.6 ms p50 and 19.5 ms p90 per request, 9.7 ms mean;
• about 105 requests/s one request at a time, and about 97 requests/s in packed batches (117 with vLLM's FlashAttention-2 build installed);
• 99.8% argmax agreement with our evaluation path on 1,782 dev questions.

Start-up: about 25 s until ready when the kernel cache is warm, about 2 min on the very first start (kernel compilation). The common CUDA graphs are captured at start; the rest are captured in the background after the model is ready.

The model is 0.8B parameters: 1.7 GB in bf16, 0.8 GB in 8-bit.

Training

qev-0.8b is a full fine-tune of Qwen3.5-0.8B-Base (all text layers and the final norm; the tied embeddings are the base's). It was trained on our own decision data, in stages:

1. Base fine-tune: two epochs, about 1.8B tokens, over a decision mix of 1.28M requests (learning rate 5e-6; DiLoCo over 16 workers).
2. Anneal on a newer data release (about 142M tokens).
3. Coherence anneal: about 52M tokens, with a consistency loss on twin sets built from our own probability laws: invariance to rephrasing and reordering, complements, partitions, bounds, monotonicity and conditioning. No benchmark code or items were used.
4. A further round on newer data with coherence twins (about 104M tokens).
5. Weight soups of the trunk tensors to keep calibration while adding consistency.
6. A gap pass (150M tokens) over data for weak skills, BH-style long hard decisions, hard yes / no and grounding negatives, order-invariance twins, and replay.
7. A 50/50 weight soup of the pre-gap and post-gap models (the first release).
8. A games, rules and spatial pass (109M tokens with replay; 27M new): code-generated game-move decisions with the facts in the question or in the state, rules written in the question, and coordinate / grid relations. The labels are computed by code; the benchmark-only games and domains above are excluded, and every row was screened against the benchmark items.
9. The release: a 50/50 weight soup of the model from step 8 and the first release. The per-type temperatures were then refit on the calibration split.

Training data and licenses

The decision data comes from public datasets and data generated for the project. Each training row records its source, its license and how its label was made. Labels are of these kinds:

By license, for the decision dataset release the training mix was drawn from (2.2M requests):

• Permissive licenses, about two thirds of the rows: CC-BY-4.0 (about 35% in total, including NVIDIA's Nemotron agentic data, reasoning-core's procedural pile, FLD and ProofWriter), CC0-1.0 / public domain (about 17%, including the CFPB complaint database and NVD), Apache-2.0 (about 8%), MIT (about 5%, including QA sets under mixed MIT / CC-BY / Apache terms) and BSD-3-Clause (2.7%). A further 1.8% is under per-source upstream terms or ODC-BY.
• Share-alike sources, about 10.5%:
  • CC-BY-SA-4.0 from MTOP (1.6%), the Schema-Guided Dialogue dataset (1.1%), SQuAD-derived QNLI (1.1%) and HotpotQA (0.2%), plus other CC-BY-SA-4.0 sources (4.3%);
  • DBpedia (CC-BY-SA-3.0, 1.1%);
  • Bitext customer-support data (CDLA-Sharing-1.0, 1.0%).
• Generated for the project, about 17%:
  • computed by programs (6.3%);
  • written by an LLM and checked by two blind solver models (8.7%);
  • other generated rows released under CC0 (2.0%).
• Model-made labels: a further 1.2% carries ratings made by a language model, under MIT.
• Research-use releases: about 0.7% comes from the LeWiDi research release, whose component datasets carry their own terms, including research-use-only terms for one of them.

Later training stages added program-generated and writer/verifier-generated rows of the same kinds. The model is released under Apache-2.0. Whether share-alike or research-use terms on a part of the training data affect a given use of the weights is for the user to assess.

Limitations

• Ordinal scores are the weakest type. Treat score outputs as soft signals.
• Long and hard inputs. Accuracy drops on long states (several thousand tokens) and on deliberately hard items. The model reads the request in one pass and does no reasoning.
• Coherence is improved, not guaranteed. Complements and marginalisation across related questions, and unit restatements such as "90 min" vs "1.5 h", are still often violated. Use per_question for multi-question requests, and do not assume that answers to related questions are mutually consistent.
• Abstention. The model rarely chooses "cannot answer" or "no tool call" options, even when it should. Give it an explicit option and check its probability rather than relying on the argmax.
• Calibration is fitted, not universal. On data far from the calibration split, probabilities can be over- or under-confident. Recalibrate on your own labelled data when probabilities drive decisions.
• Language and domain. The data is mostly English business, operations and agent-log decisions. Other languages and domains have not been evaluated.
• Options. A question can have at most 255 options. Option names matter, because the model reads them.
• Images are zero-shot. qev was trained on text only and reads images through the base model's vision tower. It is weak on board positions (connect-four and chess are near chance, tic-tac-toe 0.56) and on counting real objects (CLEVR 0.39), and it was tested on one image per request; videos are not supported. Image requests cost more tokens than a short text state (up to 196 per image by default), and their speed on a Mac has not been measured (the Linux numbers above are not representative of Apple silicon).
• Inherited behaviour. The model inherits biases of the base model and of its data, part of which was labelled by larger models. Do not use it as the only decision maker for consequential decisions about people (medical, legal, credit, employment).

Files

Provenance

The base is Qwen/Qwen3.5-0.8B-Base at revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68. qev-0.8b is the release called SG50 in our internal reports.

The MLX weights are qev-0.8b's weights in mlx-lm's Qwen3.5 layout (mlx_lm.utils.load_model loads them); q8/ holds the same model with 8-bit affine weights, group size 64. vision.safetensors holds the base model's 153 vision tensors (tower and merger) from the qev-0.8b checkpoint, values unchanged, in bf16 in the layout of qev_mlx_vision.py.

License

Apache-2.0, as is the base model, Qwen3.5-0.8B-Base. See "Training data and licenses" for the data.