OpenDecider-nano ONNX

opendecider-nano, the ~400M calibrated decision model, as ONNX for the browser (WebGPU or WebAssembly), Node, Bun and Deno. Give it a state (text or JSON) and typed questions (choice, score, noul); it returns a probability for every option, on the user's device: nothing is sent anywhere. These are the files @opendecider/web loads, and the files OpenDecider Focus runs to filter a YouTube feed (a Chrome extension on the Chrome Web Store).

Try it in your browser · Guide · GitHub

npm install @opendecider/web

import { loadNano, choice, noul, Guard } from "@opendecider/web";

const model = await loadNano(); // these files, at the revision the package pins, SHA-256 checked, then cached
const r = await model.systemOne("I was charged twice for order 1182. Please fix this today.", {
  team: choice("Which team should handle this?", { billing: "charges, refunds", tech: "bugs, outages" }),
  urgent: noul("Does this need a reply today?"),
});
r.answers.team; // { type: "choice", choice: "billing", probabilities: { billing: 0.92, tech: 0.08 }, ... }

const guard = new Guard({ model }); // the prompt guard, on the device

Files

One graph each: input_ids and attention_mask (int64, [batch, tokens]) in, logits (float32, [batch, tokens]) out, one score per token; the answer is the softmax over the scores at the [MASK] markers, one before each option (opendecider.json has the details). In q8 and q8f16 the weights are 8-bit (ONNX Runtime's MatMulNBits, block 32): weight-only, so the browser's WebAssembly gives the same logits as native ONNX Runtime to 1e-6 (fp16: within 0.01).

Same answers as the PyTorch model

Scored with native ONNX Runtime against opendecider-nano's committed answers, on every benchmark the model card reports:

On WebGPU, which computes in float16 (Chrome, Apple M4 Max, all 2,400 typed-decisions and general questions), fp16 gave PyTorch's answer on 99.45–100% of them and q8f16 on 99.3–99.5%.

As a guard against jailbreaks and prompt injection (2,438 prompts, each build at its own train-split threshold, which @opendecider/web's Guard uses):

4-bit builds were not released: they changed 2.5–4.3% of the answers. Every number: Benchmarks.

Will it run on my device?

ONNX Runtime's 8-bit WebGPU kernels are tuned for one question at a time, so for many at once (a feed, a batch) use fp16 on WebGPU. On WebAssembly fp16 is no faster than q8 and needs about 4.8 GiB: keep q8 there.

A browser with WebGPU uses it; otherwise WebAssembly. More WebAssembly threads need a cross-origin-isolated page (see the guide). Phones with little memory may not load it.

Without JavaScript

The graph runs in any ONNX Runtime. In Python, with the same prompt as the opendecider package (0.7.0 or later):

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
from opendecider.prompt import nano_ids

tok = AutoTokenizer.from_pretrained("manjunathshiva/opendecider-nano")
sess = ort.InferenceSession(hf_hub_download("manjunathshiva/opendecider-nano-ONNX", "onnx/model_q8.onnx"))
options = {"billing": "charges, refunds", "tech": "bugs, outages"}
ids, truncated = nano_ids(tok, "I was charged twice for order 1182.", "Which team should handle this?", options, 2048)
x = np.array([ids])
logits = sess.run(["logits"], {"input_ids": x, "attention_mask": np.ones_like(x)})[0][0]
z = logits[x[0] == tok.mask_token_id].astype(np.float64)
p = np.exp(z - z.max()); p /= p.sum()
dict(zip(options, p.round(4)))   # {'billing': 0.9476, 'tech': 0.0524}

Check the files yourself

The files rebuild byte for byte from the repository's packaging/onnx/ (torch 2.14.1, transformers 5.18.0, onnx 1.23.1, onnxscript 0.7.2, onnxruntime 1.30.0), from opendecider-nano at revision beeeb640f3333aec4ef78ed3b7bd0605f973a590:

python packaging/onnx/export_nano.py --out build/ --revision beeeb640f3333aec4ef78ed3b7bd0605f973a590
python packaging/onnx/quantize.py build/model.onnx build/onnx/
python packaging/onnx/make_web_tokenizer.py <opendecider-nano>/tokenizer.json build/tokenizer.web.json
shasum -a 256 build/onnx/*.onnx build/tokenizer.web.json

@opendecider/web pins this repository's revision and the size and SHA-256 of each file, and never uses a file that does not match.

Limits

• English, like opendecider-nano: evaluated in English only.
• 2,048 tokens per input; a longer state is shortened first (the answer is marked truncated).
• Not for edge functions: Cloudflare Workers and similar runtimes have far less memory than the model needs.
• opendecider-nano's own limits apply; see its model card.

License

Apache-2.0, as opendecider-nano. Built on Ettin-encoder-400m (MIT); see NOTICE. Manjunath Janardhan.