Superseded: this model is from an earlier Jeff release and is no longer updated. The current release is Jeff v1.3: mstrasser/jeff-adapter-tools. This v1.2 adapter loads only on the v1.2 base.

Jeff-Qwen3.5-0.8B-tools

Agent tool choice: picks the next tool for an AI agent to call, or says to answer directly or to ask the user for missing information.

Give it the agent's purpose, the conversation so far, the user's latest message and the agent's tool list (each tool's signature and a one-line description). It returns a probability for each tool, and for two fixed options: answer directly, or ask the user for missing information first. It picks the next step only; it does not fill in arguments. It is a LoRA adapter for Jeff-Qwen3.5-0.8B v1.2, a small open decision model: it answers in one forward pass, with no generated text to parse.

Results

Test set: Tools held-out agents, 5,157 rows. Requests to the 10% of the roughly 450 generated agents that were never trained on. It is included in this repo as test.jsonl.

Calibration error is the expected calibration error over 15 confidence bins, after each model's fitted temperature: the average gap between how confident the model is and how often it is right. Lower is better. The untrained Qwen3.5-0.8B is read out from its own scores for the option letters.

Other test sets (planned, from public data): BFCL v3, multiple and irrelevance; When2Call, test. Not measured yet.

Against Qwen3.8-27B

Here Jeff + tools answers first. Below a confidence threshold, the query is also sent to Qwen3.8-27B and the 27B's answer is used. Each adapter's threshold is the fastest one that still beats the 27B alone by at least 1 point of accuracy on the adapter's calibration rows (not the test rows).

The threshold is 0.00: on the calibration rows the adapter alone already beat the 27B by more than a point, so no query is passed on. The route is 36.5× faster than the 27B alone.

How this was measured: a fixed random sample of 300 held-out test rows, the same rows for both routes, on an Apple M4 Max (128 GB) with both models in MLX, one at a time. Qwen3.8-27B ran with 8-bit weights, prompted, with step-by-step reasoning off. Times are the mean per query over the whole sample, from prompt to answer. This is a 300-row sample, not the full test set.

Speed and memory

One request at a time on an NVIDIA RTX PRO 6000: 675 requests per setting, 75 test prompts of each of the nine adapters, interleaved. Measured for the adapters as a group, not for each adapter, so treat these as typical figures:

Jeff's speed and memory on the Mac: not measured yet.

Where it goes wrong

On the full test set:

• The commonest mistake is picking another tool from the list (training lists included deliberate near-duplicate tools, so give each tool a clear, distinct description). Next: calling a tool when the user should have been asked for missing information first. ask_user is the weakest answer.

Commonest confusions (right answer → chosen instead, rows):

Accuracy by right answer:

When to use it

• Your agent has a list of tools and must decide, at the start of each turn, which one to call next.
• You want a quick first decision with a probability, so a low-confidence turn can be passed to a larger model.
• Your tool list is your own; the adapter reads each tool's signature and description (2 to 150 tools in training).

When not to use it

• You need the tool's arguments filled in. The adapter picks one tool; it does not write the call.
• The task needs several tools in a row. The adapter chooses only the next step; ask again after each call.
• Your tool list is so long that the request would pass 8,192 tokens, the longest row in training.

How to use it

The adapter runs with Jeff's server on the Jeff-Qwen3.5-0.8B v1.2 base. One server holds the base and any number of adapters; each request picks one by name with "model": "tools".

git clone https://github.com/firelex/jeff && cd jeff
uv sync --no-default-groups --extra lora          # add --extra cuda on NVIDIA GPUs, --extra mac on Apple silicon
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B --revision v1.2 --local-dir Jeff-Qwen3.5-0.8B-v1.2
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B-tools --local-dir adapters/tools
JEFF_CHECKPOINT=Jeff-Qwen3.5-0.8B-v1.2 JEFF_ADAPTERS=adapters PORT=8765 \
  uv run --no-default-groups --extra lora jeff-serve
# add JEFF_BACKEND=mlx and --extra mac for Apple silicon

Every folder in adapters/ is served under its folder name. Add or replace an adapter while the server runs with POST /v1/adapters/reload. The server checks the base weights' checksum, so the adapter loads only on the exact v1.2 base.

from jeff import Client

tools = Client("http://localhost:8765", model="jeff-latest").with_model("tools")

Request format

State (the situation):

Questions:

• tool (choice): Which tool the agent should call next, or whether to answer directly or ask the user for missing information first. Options: answer_directly and ask_user first, word for word, then your tools as t1, t2, … with each tool's signature and a one-line description.

Rules:

• Keep answer_directly and ask_user as the first two options, with exactly the wording in the example.
• List your tools after them in a fixed order so the unchanging part of the request can be prepared in advance.
• Choose answer_directly also when the user asks for something no tool can do; the agent then explains that it cannot.
• Use the instructions below word for word; the adapter was trained mostly on them.

Example

{
  "model": "tools",
  "state": {
    "agent": "An assistant that manages the calendar and email of a small design studio.",
    "conversation": [
      {
        "role": "user",
        "text": "What's on my calendar tomorrow?"
      },
      {
        "role": "assistant",
        "text": "Tomorrow you have a client call with Harbour Books at 10:00 and a team review at 15:00."
      }
    ],
    "user_message": "Please move the team review to Friday at the same time."
  },
  "questions": {
    "tool": {
      "type": "choice",
      "instructions": "Which tool should the agent call next to handle the user's latest message? Use the conversation for context. If no tool is needed, choose answer directly. If a tool is needed but information it requires is missing, choose ask the user.",
      "criteria": {
        "answer_directly": "No tool is needed: answer the user directly from the conversation",
        "ask_user": "A tool is needed but required information is missing: ask the user first",
        "t1": "list_events(date): list the calendar events on a given day",
        "t2": "create_event(title, start, end, attendees): add a new event to the calendar",
        "t3": "move_event(event_id, new_start, new_end): move an existing event to another date or time",
        "t4": "send_email(to, subject, body): send an email now",
        "t5": "draft_email(to, subject, body): save an email as a draft without sending it"
      }
    }
  }
}

curl -s localhost:8765/v1/systemone -H 'content-type: application/json' -d @request.json

The answer holds a probability for each option of each question, the chosen option and a confidence. (A recorded response to this example is not published yet.)

Training

The replay slice mixes in some of the base model's own training data so the adapter keeps the base's general skills. The scored checkpoint is the one at the end of the epoch.

Data and provenance

Sources:

Which models wrote and checked the data. User messages were written by Qwen3.8-Max (hosted); the agents, tool lists and reworded instructions by Qwen3.8-Flash-Next (local). Labels were judged and checked by Qwen3.8-Flash and DeepSeek-V4-Flash (hosted), and a few by Qwen3.8-Flash-Next (local). Qwen3.8-Max is a hosted, closed-weight model, used through Alibaba Cloud's DashScope API to finish in time. Counts are on the 39,203 training rows; one row can pass through several models, so the counts do not add up to the total.

Training data not published. The test set is included in this repo as test.jsonl, and the calibration set (2,195 rows, the rows the threshold for the comparison with Qwen3.8-27B is chosen on) as calibration.jsonl.

Test set: test.jsonl, 5,157 rows: Requests to the 10% of the roughly 450 generated agents that were never trained on. It is byte for byte the set the adapter was scored on.

The terms of the hosted model providers are being checked for training and publication use.

Quality review. An independent reviewer checked the training, calibration and test files for shortcuts, duplicates, leaks between splits and junk before training. The report is at jeffhub.ai/adapters/tools/qa-report.

Limitations

• English only.
• Tied to the Jeff-Qwen3.5-0.8B v1.2 base. It will not work on any other base or version; the server refuses it if the base weights' checksum does not match. A v1.3 long-term-support base is coming, and the adapters will be retrained on it.
• Jeff chooses between the options you give it. It does not write text or reason in several steps.
• Calibration was fitted on this adapter's own calibration rows. On very different data, check it again.
• Everything listed under When not to use it above.

Links

• Adapter page: jeffhub.ai/adapters/tools
• Code and server: github.com/firelex/jeff
• Base model: mstrasser/Jeff-Qwen3.5-0.8B (revision v1.2)

Jeff is an independent project. It uses the same request format as Jev but is not affiliated with or endorsed by TypeSafe, the makers of Jev.