Superseded: this model is from an earlier Jeff release and is no longer updated. The current release is Jeff v1.3: mstrasser/jeff-base.

Jeff-Qwen3.5-0.8B

Community preview: v1.2 (1 October 2026), with nine LoRA adapters. Catalogue, results and docs on jeffhub.ai.

Put Jeff in front of your 27B

Let a strong local model such as Qwen3.8-27B write and plan, and let Jeff make the quick decisions in front of it: Jeff answers first, and only when it is unsure does the query go on to the 27B. Across 8 adapters*, with the same test rows both ways on an Apple M4 Max:

On the five decisions an inbox agent makes for every message alone: 87.7% → 95.7%, 39× faster. Jeff wins outright on 8 of the nine adapters and draws on grounding (96.3% against 96.7%, at 20× the speed). The 27B ran in 8-bit with step-by-step reasoning off; each task used a fixed sample of held-out rows (300; 500 for emotion and legal-clauses).

*Emotion is left out of the averages: picking the single strongest of 27 emotions (or neutral) in short comments is hard even for people, and the human labels often disagree. Jeff + adapter scores 60.6% there against the 27B's 35.6%. Including it, the average across all nine adapters is 91.4% against 80.9%.

Per-task results and the method are on jeffhub.ai and in the GitHub README.

v1.2 is more honest, not smarter. We found that v1.1's training data overlapped some of our own test sets, and that parts of it let a model score by answer order or answer length instead of by content. v1.2 is trained on cleaned data. Its benchmark score is about the same (78.7% against 79.1%), and its score on our long-document test drops from 82.8% to 66.1%, because v1.1's figure was inflated by leaked test documents. v1.2 is also the base for nine small LoRA adapters (legal clauses, support intents, emotion, spam, grounding, prompt-injection guard, tool choice, voice navigation, ticket triage) that one server loads side by side. Details: What changed in v1.2. v1.1 stays available as revision v1.1.

The Jeff models are fine-tunes of Qwen3.5 and Gemma 4 for zero-shot classification: small, fast decision models you slot into your code. You describe a situation and list the options in plain words; Jeff returns a calibrated probability for each option from a single forward pass. No generated text, no parsing: about 22 ms per decision on an RTX PRO 6000 and 28 ms on an Apple M4 Max (MLX).

Zero-shot means the options can be anything: support queues, user intents, moderation labels, voice commands, game moves. Your categories don't need to appear in the training data; you describe them, and Jeff picks.

What it is, and what it isn't. These are very small models. They make extremely fast, well-calibrated judgement calls between options, and they slot easily into your local code. On benchmarks they approach, and sometimes beat, Jev; but at this size their reasoning won't match Jev's, which runs on a much larger model. If zero-shot accuracy isn't good enough for your purposes, a short fine-tune on your own examples takes you much further: our voice-navigation fine-tune moved held-out accuracy from 31.7% to 95.8% in under half an hour on one GPU.

Built entirely on local hardware. Training on one RTX PRO 6000 workstation GPU (the 0.8B trains in about 2 hours, the 2B in about 3.5), all synthetic training data of the base model written by an open model (Qwen3.8-Flash-Next) on two DGX Sparks, testing on a MacBook. No cloud GPUs, and no closed model was used as a teacher for the base model; a closed model was used only to spot-check the quality of a sample of the synthetic data. Some public data sets in the mix contain text their authors generated with closed models (for example the model responses in RAGTruth). Part of the adapters' data was written by a hosted model; see LoRA adapters on v1.2.

Independent project. Jeff uses the same request format as Jev, but it is not affiliated with or endorsed by TypeSafe, the makers of Jev. Our training code starts from the open-source AutoJev recipe.

What changed in v1.2

v1.2 is a new training run of the 0.8B on cleaned data, with the same recipe as v1.1 (one epoch, final checkpoint, one fitted temperature): 284,747 training questions instead of v1.1's 322k. The 2B was retrained on the same data (v1.2; its card has its results). What changed in the data:

• Leaks removed. A stricter check (exact states at any length, option texts unique to one test item, near-duplicate passages, and the upstream record behind each item: the same contract, document or source sentence) found training questions that overlapped our test sets: 5,279 for the long-document test, 4,358 for the first long-list test, 203 for the voice test, 66 for the benchmark panel and 4 for the calibration file. All are gone; the check now runs before every training run, and the v1.2 mix passes it with no overlap. The biggest single cause: MAUD (merger agreements) splits its data by question, not by contract, so all 153 contracts in our document test were also in training. Every MAUD training question (4,309) is removed.
• Shortcuts removed. In one family of synthetic answer comparisons the longer answer was correct 85% of the time; it is now 50%. Synthetic yes/no families are now 50/50. We evened out which answer letter is correct, closed gaps in letter sequences, stopped the number of options from giving away "none of these" answers, and re-sampled two synthetic families where the correct answer was too often the longest or the shortest. Public data sets keep their natural label rates.
• Voice navigation moved out. The 5,703 voice-navigation questions left the base mix and went into their own adapter (nav). The voice score below is therefore zero-shot for v1.2.
• Made-up long lists were cut from 15,525 questions to 5,001; the MASSIVE and CLINC150 long lists are all kept.

Results, v1.1 final checkpoint against v1.2 final checkpoint:

Per benchmark (v1.1 → v1.2): BBH 64.9 → 63.2, Financial PhraseBank 95.7 → 96.3, JudgeBench 63.1 → 60.9, RAGTruth 85.6 → 85.5, WinoGrande 69.0 → 68.7. We read v1.2 as level with v1.1 on the panel and slightly less well calibrated.

So why release it? Because its numbers mean what they say, and because it is the base the adapters are trained on. If you rely on v1.1 for long contracts, expect v1.2's figure, not v1.1's.

v1.1 is still available: add --revision v1.1 to hf download (or --revision v1.0 for v1.0).

LoRA adapters on v1.2

A LoRA adapter is a small add-on (about 41 MB for these) that tunes Jeff for one job. The server loads the base once and any number of adapters beside it; each request picks one by name in its "model" field, and adapters can be added or replaced while the server runs (POST /v1/adapters/reload). Each adapter has its own card with its data, how the data was made, and its results against the untrained Qwen3.5-0.8B and against v1.2 alone:

Accuracy on each adapter's full held-out test set, which is included in its repository.

Data disclosure for the adapters: unlike the base model, part of the training data of ground, guard, tools, nav and triage was written by Qwen3.8-Max, a hosted, closed-weight model, through Alibaba Cloud's DashScope API, and part was checked by hosted Qwen3.8-Flash. The other four adapters use public data only. Each card gives the exact split by model.

What changed in v1.1 (the Qwen models)

• Long lists. The v1.0 models never picked an option past the 26th, because no training question had more than 19 options (thanks to @puhuk for the report, #1). v1.1 adds 32,000 training questions with 20 to 254 options and accepts up to 254. On our long-list test (lists of 20 to 254 items) Jeff-Qwen3.5-0.8B goes from 40.3% to 94.7%, and Jeff-Qwen3.5-2B scores 95.2%.
• Final checkpoint. v1.1 publishes the checkpoint at the end of the single epoch. Choosing the checkpoint with the lowest development loss, as v1.0 did, picked an early, less settled checkpoint in the new runs and cost 1–2 points.
• Calibration is better: calibration error 0.049 → 0.021 for the 0.8B and 0.028 → 0.026 for the 2B. JevBench's hard tier: 47.6% → 46.7% for the 0.8B, 53.3% → 57.1% for the 2B.
• Benchmarks: the 0.8B is unchanged at 79.1%. The 2B goes from 83.1% to 82.0%, mostly on JudgeBench (64.6% → 59.4%), a small benchmark where models of this size sit close to chance; so the 2B no longer edges past Jev's published 83.0%.
• Training data also adds the training splits of MASSIVE and CLINC150 (as full intent lists), so results on those two data sets are no longer zero-shot, and more synthetic data from our local teacher.
• v1.0 is still available: add --revision v1.0 to hf download. The game results below were measured with v1.0.
• Jeff-Gemma4-E2B was not retrained and stays at v1.0.

What it does

{
  "model": "jeff-latest",
  "state": {"voice_transcript": "open the engagment leter", "current_screen": "Deal overview"},
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "Which of these does the user want?",
      "criteria": {"1": "Engagement letter", "2": "Inbox", "3": "Deal settings"}
    }
  }
}

The answer is a probability per option ({"1": 0.94, "2": 0.03, "3": 0.03}), the chosen option and a confidence. Three question types: choice (pick one of up to 254 options with the Qwen models, 26 with Jeff-Gemma4-E2B), noul (yes/no, returned as a probability) and score (a point on a scale you describe). Several independent questions in one request are answered together.

Benchmarks

4,599 questions from five public benchmarks, plus JevBench's public hard tier (105 items, scored separately):

The Qwen models are v1.2, Jeff-Gemma4-E2B v1.0.

Bold: the winner of Jeff against Jev in each row: a Jeff score above Jev's published figure, or Jev's figure where it beats every Jeff model. Bold italic: AutoJev-27B where it is the best of all models in the row; it is shown for reference, since the head-to-head comparison is with Jev. The published Jev and AutoJev figures were measured on a different sample of the same benchmarks. Jeff's overall score comes from classification and grounding (Financial PhraseBank, RAGTruth), where it matches or beats the large models; on the reasoning-heavy benchmarks (BBH, JudgeBench, JevBench) it stays well below them, as you would expect at this size.

Games: a zero-shot test

To test zero-shot performance on tasks unlike anything in the benchmarks, we had Jeff play three games. Games aren't the ideal zero-shot test, since a game's state isn't typical unstructured data; but they are a common, and fun, way to test a System 1 model. Each turn, the code describes the situation and the legal moves in words, and the model picks one. The options state what each move leads to (Frogger: "you would be hit by a car and lose a life"; Doom: "the nearest monster is a little to your left"), but never which move is right. Each result is 20 episodes, seed 1234; ▶ opens a video of the run's first episode. The games were measured with v1.0 of the Jeff models.

Jeff-Qwen3.5-0.8B playing, zero-shot (the bold row in the table below):

Jeff-0.8B decides in 29–49 ms per move on an M4 Max; Jev's published Doom run took 212 ms per call over its API. The two times were not measured on the same hardware.

Speed and size

v1.2 has the same architecture and size as v1.0 and v1.1; these timings were measured with v1.0. Median time per decision over the same 200 benchmark questions (about 200 input tokens each), one question at a time, from raw text to probabilities:

How to use

git clone https://github.com/firelex/jeff && cd jeff
uv sync --no-default-groups   # serving only; add --extra cuda on NVIDIA GPUs (fast kernels), --extra mac on Apple silicon
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B --local-dir Jeff-Qwen3.5-0.8B
JEFF_CHECKPOINT=Jeff-Qwen3.5-0.8B PORT=8765 uv run --no-default-groups jeff-serve                        # NVIDIA or CPU
JEFF_BACKEND=mlx JEFF_CHECKPOINT=Jeff-Qwen3.5-0.8B PORT=8765 uv run --no-default-groups --extra mac jeff-serve       # Apple silicon
curl -s localhost:8765/v1/systemone -H 'content-type: application/json' -d @request.json

• Reason in code, decide with Jeff. State each option's consequence; don't ask Jeff to forecast.
• Use short option keys and descriptive text: {"1": "Engagement letter"}, not long IDs, which cost time and add nothing.
• Fine-tune for your domain. A voice-navigation fine-tune on ~11k app-specific examples took held-out accuracy from 31.7% to 95.8% in one epoch.

Training

• Recipe: full-weight supervised fine-tuning, one epoch (learning rate 5e-6 for the 0.8B, 1e-5 for the 2B), batches of 256, cross-entropy over the option letters, then one scalar temperature fitted for calibration. From v1.1 we publish the checkpoint at the end of the epoch; the benchmark panel is never used for selection or tuning. v1.2 trained for 1,113 steps.
• Data (284,747 questions in v1.2; 322k in v1.1; 271k in v1.0): public datasets converted to decisions (entailment, QA, sentiment, safety, fact verification and more), the training splits of WinoGrande (40k) and RAGTruth (15k), long real documents (ContractNLI, CUAD, ConditionalQA), 10k code-built probability questions with exact answers, ~50k synthetic questions written and checked by a local teacher model (Qwen3.8-Flash-Next on DGX Sparks), hijack attempts planted in 3% of questions, "none of these" options added to 5%, and (v1.1) 32k long-list questions with 20–254 options, from code and from the MASSIVE and CLINC150 training splits. v1.2 removes the voice-navigation questions, all MAUD questions and every overlap with a test set, and cuts the made-up long lists to 5,001 (see What changed in v1.2).
• Leak filter (v1.2): every training question is checked against the benchmark panel, JevBench, the long-document test, the voice test, both long-list tests, the chess tests and the development and calibration files: exact states at any length, option texts unique to one test item, near-duplicate passages, and the upstream record (MASSIVE utterance, MAUD or CUAD contract, ContractNLI document, ConditionalQA page). The v1.2 mix has no overlap. (v1.1's filter checked only the panel and JevBench and removed 50 near-duplicates.)
• Disclosure: at least half of each training family is written in the benchmark panel's layout conventions (how states, questions and options are formatted). No panel item is used in training, but matching the format helps the score.
• Hardware: one RTX PRO 6000 (96 GB) for training, DGX Sparks for the teacher, and an M4 Max for local tests.

Caveats

• Option limits. The Qwen models are trained on lists of 20 to 254 options and accept up to 254. Jeff-Gemma4-E2B is still v1.0: its training never had more than 19 options, so it only picks reliably among up to 26, and the server refuses longer lists for it; shortlist first. Thanks to @puhuk for the report (#1).
• Small models don't reason. Expect fast, calibrated choices between the options you describe, not multi-step reasoning. At 0.8B–2B parameters this holds for every model, not just Jeff.
• Jeff-2B is a weaker game player than Jeff-0.8B. The untrained 2B already appears more risk-averse than the untrained 0.8B (in Doom it prefers turning away from the nearest monster; in Pac-Man it survives much longer but hesitates), and our training seems to have made that worse: Jeff-2B reverses direction in Pac-Man 3.5× as often as the untrained 2B. This needs more investigation.
• Benchmark scores don't predict game play. The untrained Gemma 4 E2B beats the untrained Qwen models on the benchmarks (62.5% against 45–47%) yet plays the games worst: it makes the right move most of the time but not reliably, and in a real-time loop the occasional wrong call compounds. Training fixed its Pac-Man (3.2 → 53.2 pellets) but not its Doom or Frogger.
• Prompts matter. The game results depend on options that state consequences in words. Jev's own Doom prompt (a raw bearing number plus an aiming rule) does not work for any of our models, trained or untrained.
• Different samples. The published Jev and AutoJev numbers were measured on a different sample of the benchmarks.
• Long documents: v1.2 scores 66.1% on our long-document test (MAUD 51.4%, CUAD 89.9%, ContractNLI 85.8%, ConditionalQA 59.9%). Merger-agreement questions (MAUD) are hard for a model this size without having seen the contracts.
• English and text only. Calibration is fitted on our development data; recalibrate for a very different domain.

Licence and data

Weights: Apache 2.0 (from Qwen3.5-0.8B by the Qwen team, Alibaba Cloud). The training data mixes datasets under various licences, listed with their sources in docs/data-sources.md. We release the weights and code, not the training data; some sources are share-alike (CC BY-SA).