mohit67890/imajev-9b

Decisions for real-world cases.

Small open models that read the photos, records and text a business already has and answer in the options you set, with a probability on each and an explicit can't tell. Your system acts when it is sure and hands the rest to a person.

imajev-9b is the largest size of the family · other sizes: imajev-2b · imajev-4b
Live demo · Website · Code and results · Technical report

_{Ranks of the family's imajev-4b. This 9B has no official text-board entry.}

Independent results (screenshots of the official leaderboards, 28 Sep 2026)

_{The family's imajev-4b: JevBench v1.4.2.2, scored 27 Sep 2026 (board). Image JevBench v0.1.3, released 28 Sep 2026: #1 of 49, 76.39, ahead of Jev-Omni (board). DecisionBench (eng, v1): #3 of 56 models, 79.65, ahead of GLM-5.3 Flash (320B), Jev 1.13, DeepSeek V4.1 Flash (552B) and GPT-5.6 Luna; the two above are the benchmark team's own models (leaderboard).}

Built by Mohit Garg. I build decision systems like this for operations, support and back-office teams:
refunds, returns, ticket routing, document and photo checks. Work with me → mohitgarg.ai

imajev-9b is the largest size: it automates the most decisions and is strongest on knowledge-heavy text (MMLU 79.2%, measured on the previous imajev-9b).

2026-09-26: the 4B tier moved to a new adapter (phase 3: ImajevBench 83.9%, DecisionBench full suite 79.7%, JevBench hard 72.1% as shipped) and is now the family's best size on ImajevBench; this 9B adapter is unchanged and remains the previous generation. See https://huggingface.co/mohit67890/imajev-4b.

What sets it apart

• A photo read against your record. Checks a photo against your own fields and names the one that is wrong. Trained on 72k photo-vs-record and two-photo decisions.
• Two photos, one decision. A reference and a target in the same request: shipped against returned, a known-good part against the one on the line.
• A trained can't tell. Every answer carries a probability for unknown, so the app can stop instead of guessing.
• Open, small and local. Apache-2.0, MLX on a Mac or PyTorch on one GPU; photos and customer data never leave your network.
• Jev's contract, now with images. TypeSafe's Jev request and response (POST /v1/systemone), plus images, unknown_probability and abstained. Jev itself is text-only and hosted; its state limit (32k tokens) is larger than imajev's (32 KB).

One request, every answer typed

The exact script we ran against imajev-4b and its output (rounded, usage shortened); 1.15 s on a Mac Studio (four option orders averaged, calibration file applied). Swap the adapter for this size and the request is unchanged.

import json, requests

URL = "http://127.0.0.1:8765/v1/systemone"

listing = {
    "title": "Men's suede boat shoes",
    "color": "red",
    "product_type": "shoe",
}

questions = {
    "contradicted_field": {
        "type": "choice",
        "instructions":
            "Which field of `listing` does this photo contradict?",
        "criteria": {
            "listing.color": None,
            "listing.product_type": None,
            "none of these": "the photo agrees with every field",
        },
    },
    "color_matches": {
        "type": "noul",
        "instructions":
            "The product in the photo matches `listing.color`.",
    },
    "type_matches": {
        "type": "noul",
        "instructions": "The photo shows the kind of product "
                        "given in `listing.product_type`.",
    },
}

request = {"state": {"listing": listing}, "questions": questions}
with open("listing.jpg", "rb") as photo:
    r = requests.post(URL, files={"image": photo},
                      data={"request": json.dumps(request)})
print(json.dumps(r.json(), indent=2))

Result

{
  "model": "imajev-4b",
  "answers": {
    "contradicted_field": {
      "type": "choice",
      "choice": "listing.color",
      "probabilities": {
        "listing.color": 0.95,
        "listing.product_type": 0.006,
        "none of these": 0.043
      },
      "confidence": 0.919,
      "unknown_probability": 0.007,
      "abstained": false
    },
    "color_matches": {
      "type": "noul",
      "noul": 0.082,
      "unknown_probability": 0.022,
      "abstained": false
    },
    "type_matches": {
      "type": "noul",
      "noul": 0.989,
      "unknown_probability": 0.004,
      "abstained": false
    }
  },
  "usage": {
    "total_ms": 1152.9,
    "input_tokens": 224
  }
}

Automate what is clear, route the rest

imajev-9b on the 279 ImajevBench test questions (photos, records and text; 21 whose honest answer is can't tell), raw probabilities, scored with the benchmark's own rule:

The rest go to a person. The benchmark is built to be hard; measure on a few hundred of your own cases before choosing a threshold. Other sizes at 90%: 2B 38% automated at 95.3% right, 4B 58% at 94.5%, 9B 70% at 91.8%.

How it was made

About a million training decisions across the family, in four stages, for about $676 of rented GPU time for the whole project. The 9B was trained on the 504k human-labelled decisions, then on about 23k hard questions kept only when open-weight teachers agreed, then a soft-target continuation on 39,515 rows carrying Qwen3.6-35B-A3B's full probability distributions (with the strict slice of the Eikos decisions set (caiovicentino1/eikos-decisions, CC-BY-4.0; attribution and per-source licences in docs/eikos-decisions-usage.md) and 5k replayed image decisions). The shipped adapter is the weight-space average of two adapters: the hard-question adapter and that continuation. It skipped stage 2 because it produced those labels for the 2B and 4B. Every teacher is open-weight; no Jev outputs, paid-API outputs or JevBench items were used.

This adapter

This repository holds the 9B adapter, the quality tier. It is a LoRA (rank 16, alpha 32) on the language layers of Qwen3.5-9B (revision c2022362) plus a 255-code decision readout, in PEFT format at the root and in MLX format under mlx/; the weights are the element-wise average (0.5 / 0.5, LoRA matrices and readout) of the hard-question adapter and its soft-target continuation. Code, server and evaluation harness: https://github.com/mohit67890/imajev. Other tiers: https://huggingface.co/mohit67890/imajev-4b (recommended default), https://huggingface.co/mohit67890/imajev-2b (latency).

Which size? On our measurements the 4B is within noise of the 9B on ImajevBench (82.4% vs 82.1%; a paired test on the previous versions gave p = 1.0) and one item ahead on JevBench hard (70.3% vs 69.4%), at roughly half the memory. Pick the 9B for knowledge-heavy text questions and when memory is not a constraint; otherwise start with the 4B.

Technical specification

Training path. One trainer for every stage (PyTorch + PEFT): cross-entropy on the readout logits (soft targets where a record carries a distribution), AdamW with weight decay 0, linear warm-up then cosine decay to 10% of the peak rate, gradient clipping 1.0, seed 0, 4 GPUs. Gradient checkpointing in stages 3 and 4 only. The soft-target stage adds a rationale loss (weight 0.3, at most 192 tokens) and permutes the options of every question.

Stage 1 crashed eight times in its first 980 steps (a diagnostic timer in the trainer, since removed) and resumed from 20-step checkpoints; no data was skipped.

Data this size saw.

• Stage 1: 504,000 decisions from 36 licence-admitted sources (15 text, 21 image), including 4,000 photo-vs-listing contradictions.
• Stage 2: skipped; the 9B (after stage 1) labelled the stage-2 data for the 2B and 4B.
• Stage 3, round 1: 14,112 training records: kept teacher questions (9,368 of 13,386 kept on two-answerer agreement) plus the training share of 8,532 human reasoning items from 10 licensed sets.
• Stage 3, round 2: 7,812 training records: 3,598 new (4,852 of 8,097 kept on three-answerer agreement) + 4,214 replayed from round 1.
• Stage 4: 39,515 records: the stage-3 teacher questions relabelled with Qwen3.6-35B-A3B's probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions.

Compute. $499.07 of rented GPU time on RunPod through stage 3 plus about $177 for stage 4 (8×H100, 6 h 20 min, all three sizes): about $676 for the whole project, every run included.

Full specification: https://github.com/mohit67890/imajev/blob/main/docs/technical-specification.md

Results (2026-09-24; every number reproducible from the code repository's results/)

Our pre-registered test against the untuned base model on ImajevBench (paired cluster sign-flip over 89 evidence clusters): the shipped imajev-9b vs the untuned Qwen3.5-9B, +5.4 points (229 vs 214) [−1.2, +12.4], p = 0.131, not significant at 0.05 (36 discordant clusters). The previous imajev-9b (previous adapter) gave +6.1 [+0.0, +12.7], p = 0.074, also not significant; the earlier version of imajev-9b the test was registered with gave +4.3 points, p = 0.031. We report all three. JevBench is text-only; the image capability shows only on ImajevBench and in use. The official JevBench board adds 308 sealed items and a four-axis score run by its maintainer; this 9B has no official text-board entry. The family's imajev-4b is #1 of 91 there (JevBench v1.4.2.2, scored 27 Sep 2026; https://benchmarkheaven.com/jev-models).

Older panels were measured only on an earlier version of imajev-9b, before the hard-question stage, and are not repeated here as current numbers: held-out photo sources 55.9%, real two-image pairs 26.6% (see results/imajev-9b/).

Calibration

calibration.json (schema 1.1) applies one temperature (1.748) to every question type × option-count bucket, fitted by negative log-likelihood on 150 template-generated JevBench-style items (none from JevBench), also used to pick checkpoints. Temperature scaling never changes an answer, only its probability. unknown offsets are 0: bounded offsets were tested and changed no panel by more than 0.1 points.

Checked through the released server (ECE, uncalibrated → with calibration.json); the off-distribution rows were measured on the previous imajev-9b with its own temperature (2.19) and have not been re-run for this version:

Pooled ECE over all 231 public JevBench items in the served configuration: 0.050. An ECE-fit temperature would lower the hard-tier ECE but raise the pooled ECE, as for the 4B and 2B, so the NLL fit is shipped.

On photo-only verification the raw probabilities are already calibrated and the temperature over-softens them; if your traffic is mostly photo-against-record checks, serve without --calibration or fit your own temperature on a held-out sample.

How to use

git clone https://github.com/mohit67890/imajev && cd imajev
python3.11 -m venv .venv && . .venv/bin/activate
pip install -e ".[serve,mlx]"            # Apple silicon;  elsewhere: pip install -e ".[serve,torch]"
python scripts/download_model.py --model 9b
hf download mohit67890/imajev-9b --local-dir adapters/imajev-9b
# Mac (MLX)
PYTHONPATH=src:scripts python scripts/playground/server.py --model-bundle artifacts/model-qwen9b.json \
  --adapter adapters/imajev-9b/mlx --calibration adapters/imajev-9b/calibration.json --model-name imajev-9b --port 8767
# Linux / CUDA (PyTorch + PEFT)
PYTHONPATH=src:scripts python scripts/playground/server.py --backend torch --model-bundle artifacts/model-qwen9b.json \
  --adapter adapters/imajev-9b --calibration adapters/imajev-9b/calibration.json --model-name imajev-9b --port 8767

Then POST /v1/systemone with a Jev-shaped request (state, questions, optional images). --rotations 4 averages four option orders; the numbers above use it (for this version it changed no JevBench hard item, at ~3× latency). The 9B needs ~19 GB resident; do not keep it and another model loaded on the same Mac.

Training

• Base: Qwen3.5-9B (Apache-2.0). LoRA r16/α32 on every language-model projection including the DeltaNet projections; 255 single-token option codes read at the decision position through a float32 readout head; vision tower frozen.
• Licence-checked decisions (the 9B's first stage): one epoch over 504,000 human-labelled image and text decisions from licence-verified sources, lr 2e-4, 4×H200.
• Hard-question stage: 2 epochs, lr 3e-5, on 17,898 hard typed questions (documents written by Qwen3.6-27B, kept only when two answerers of different families, Qwen3.6-27B thinking and gpt-oss-20b, agree) plus licence-verified human reasoning sets. JevBench hard went from 42.3% to 67.6%: the first-stage recipe had erased the base model's reasoning (base 64.9%), and this data restored it.
• Last part of the hard-question stage: 2 epochs, lr 2e-5, on 9,066 rows: 4,852 new questions kept only when three open-weight answerers agree unanimously (adding Qwen3.6-35B-A3B thinking) plus 30% replay of the earlier hard questions. Selected on held-out dev rows.
• Soft-target stage: 2 epochs, lr 2e-5, on 39,515 rows: the hard questions relabelled with Qwen3.6-35B-A3B's full probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions; soft-target cross-entropy plus a rationale loss (0.3) with option permutation. The shipped adapter is the weight-space average of the previous adapter and this stage's best checkpoint (0.5 / 0.5, LoRA matrices and readout): the checkpoint alone passes every gate but falls to 66.7% on JevBench hard.

Data and licence posture

• Adapter, readout and code: Apache-2.0. Base model Qwen3.5-9B: Apache-2.0.
• No JevBench items (8-gram contamination lint against the public files), no outputs from Jev, and no outputs from any paid API were used in training. All teacher models are open-weight.
• Image and text sources carry per-source licence receipts in the code repository; 16 of the 21 image sources from stage 1 are used under their annotation licences only, with photos under upstream terms (not redistributed); for abo, vizwiz, vizwiz_quality and defects the grant covers the images too.

Limitations

• Single pass, no reasoning at inference: multi-step arithmetic and answer-quality judging trail reasoning models (a frozen Qwen3.6-35B-A3B with thinking scores 97.3% on JevBench hard, at seconds and thousands of tokens per decision). The gap to JevK5 is concentrated in judge-style items.
• Over-confident without calibration.json; fit your own temperature for unfamiliar domains.
• Not better than the 4B on ImajevBench; the last part of the hard-question stage lowered the score on our reasoning dev set (also used for checkpoint selection) by 1.8 points, the soft-target checkpoint recovers it to 68.9%, and the shipped average was not measured on that set.
• At most two images, 32 KB state, 254 options, 8 questions per request. English only. No free text.

Intended use

Typed decisions inside applications: verification of a photo against a record, routing, extraction into fixed option sets, abstention when evidence is missing. Not a safety classifier, not a certificate of correctness, and not for decisions about people without human review.

Citation

@software{imajev2026,
  author = {Garg, Mohit},
  title = {imajev: an open Jev-style typed decision model family for images and text},
  year = {2026},
  url = {https://github.com/mohit67890/imajev}
}

ImajevBench, the photo-and-text benchmark released alongside: https://huggingface.co/datasets/mohit67890/imajev-bench.