Superseded: this model is from an earlier Jeff release and is no longer updated. The current release is Jeff v1.3: mstrasser/jeff-adapter-triage. This v1.2 adapter loads only on the v1.2 base.

Jeff-Qwen3.5-0.8B-triage

Support ticket triage: routes a support ticket or email to a team and scores its urgency, sentiment and need for a person.

Give it a ticket, email or chat message and a one-line description of each of your teams. In one pass it returns four answers: which team should handle it (or other), how urgent it is (five levels), how the customer feels (five levels), and whether a person needs to act on it now. Your team list is your own. It is a LoRA adapter for Jeff-Qwen3.5-0.8B v1.2, a small open decision model: it answers in one forward pass, with no generated text to parse.

Results

Test set: Triage held-out organisations, all questions, 7,256 rows. Messages to the 10% of the roughly 300 generated organisations that were never trained on, scored on all four questions (route, urgency, sentiment and whether a person is needed). It is included in this repo as test.jsonl.

Calibration error is the expected calibration error over 15 confidence bins, after each model's fitted temperature: the average gap between how confident the model is and how often it is right. Lower is better. The untrained Qwen3.5-0.8B is read out from its own scores for the option letters.

Other test sets (planned, from public data): typed-decisions, customer_service; Banking77, test split. Not measured yet.

Against Qwen3.8-27B

Here Jeff + triage answers first. Below a confidence threshold, the query is also sent to Qwen3.8-27B and the 27B's answer is used. Each adapter's threshold is the fastest one that still beats the 27B alone by at least 1 point of accuracy on the adapter's calibration rows (not the test rows).

The threshold is 0.00: on the calibration rows the adapter alone already beat the 27B by more than a point, so no query is passed on. The route is 59.1× faster than the 27B alone.

How this was measured: a fixed random sample of 300 held-out test rows, the same rows for both routes, on an Apple M4 Max (128 GB) with both models in MLX, one at a time. Qwen3.8-27B ran with 8-bit weights, prompted, with step-by-step reasoning off. Times are the mean per query over the whole sample, from prompt to answer. This is a 300-row sample, not the full test set.

Speed and memory

One request at a time on an NVIDIA RTX PRO 6000: 675 requests per setting, 75 test prompts of each of the nine adapters, interleaved. Measured for the adapters as a group, not for each adapter, so treat these as typical figures:

Jeff's speed and memory on the Mac: not measured yet.

Where it goes wrong

On the full test set:

• Almost all errors are on the two five-level scores (urgency and sentiment), and almost all of those are one level off: 3 against 4, 1 against 2, 2 against 3. The middle levels are the weakest. Routing and the "needs a person" flag are much stronger.
• Per-answer numbers mix urgency and sentiment, because both use levels 0 to 4. A per-question breakdown is not measured yet.

Commonest confusions (right answer → chosen instead, rows):

Accuracy by right answer:

When to use it

• You receive tickets, emails or chat messages and need a team, an urgency and a "does a person need to see this?" flag for each.
• Your teams are described in a sentence each; the adapter reads the descriptions, so your team list can be your own (up to 40 teams in training).
• You want all four answers from one request.

When not to use it

• You need the reply written for you. Jeff chooses between options; it does not generate text.
• The decision depends on facts outside the message, such as order history or account status. Put them in the state, or decide in code.
• Messages mostly arrive in languages other than English. Training had some European-language messages, but most are English.

How to use it

The adapter runs with Jeff's server on the Jeff-Qwen3.5-0.8B v1.2 base. One server holds the base and any number of adapters; each request picks one by name with "model": "triage".

git clone https://github.com/firelex/jeff && cd jeff
uv sync --no-default-groups --extra lora          # add --extra cuda on NVIDIA GPUs, --extra mac on Apple silicon
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B --revision v1.2 --local-dir Jeff-Qwen3.5-0.8B-v1.2
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B-triage --local-dir adapters/triage
JEFF_CHECKPOINT=Jeff-Qwen3.5-0.8B-v1.2 JEFF_ADAPTERS=adapters PORT=8765 \
  uv run --no-default-groups --extra lora jeff-serve
# add JEFF_BACKEND=mlx and --extra mac for Apple silicon

Every folder in adapters/ is served under its folder name. Add or replace an adapter while the server runs with POST /v1/adapters/reload. The server checks the base weights' checksum, so the adapter loads only on the exact v1.2 base.

from jeff import Client

triage = Client("http://localhost:8765", model="jeff-latest").with_model("triage")

Request format

State (the situation):

Questions:

• route (choice): Which team should handle the message. If it covers several issues, the team for the most important one. Options: other first (Not for any of these teams), then your teams as k1, k2, … with one line each on what the team handles.
• urgency (score): How urgently the message needs a response. Options: Five levels, lowest first, from No action needed to Handle immediately.
• sentiment (score): How the customer feels. Options: Five levels, from Very negative, angry or distressed to Very positive.
• needs_human (noul): Whether a person must act on it now rather than an automatic reply (escalating complaints, legal or safety issues, requests an automatic system cannot resolve).

Rules:

• Ask all four questions in one request; they are answered together.
• Keep other as the first route option, word for word, and list the teams after it in a fixed order so the unchanging part of the request can be prepared in advance.
• Use the instructions below word for word; the adapter was trained mostly on them.

Example

{
  "model": "triage",
  "state": {
    "company": "A mid-sized online furniture retailer shipping across Europe.",
    "channel": "email",
    "message": "Subject: Crushed wardrobe\n\nHi, the wardrobe I ordered (order 55120) arrived today and the box was crushed. Two doors are split. I want a replacement or my money back. This is the second time.\n\nAnna"
  },
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which team should handle this message? If it covers several issues, choose the team for the most important one.",
      "criteria": {
        "other": "Not for any of these teams",
        "k1": "Deliveries: late, lost or damaged deliveries, and delivery bookings",
        "k2": "Refunds and payments: refunds, charges and invoices",
        "k3": "Assembly service: booking and complaints about assembly",
        "k4": "Account access: logins, passwords and account details"
      }
    },
    "urgency": {
      "type": "score",
      "instructions": "How urgently does this need a response?",
      "criteria": [
        "No action needed",
        "Can wait a few days",
        "Handle within a day",
        "Handle within hours",
        "Handle immediately"
      ]
    },
    "sentiment": {
      "type": "score",
      "instructions": "How does the customer feel?",
      "criteria": [
        "Very negative, angry or distressed",
        "Negative",
        "Neutral",
        "Positive",
        "Very positive"
      ]
    },
    "needs_human": {
      "type": "noul",
      "instructions": "Does this need a person to act on it now, rather than an automatic reply? Answer yes for complaints that could escalate, legal or safety issues, or requests an automatic system cannot resolve."
    }
  }
}

curl -s localhost:8765/v1/systemone -H 'content-type: application/json' -d @request.json

The answer holds a probability for each option of each question, the chosen option and a confidence. (A recorded response to this example is not published yet.)

Training

The replay slice mixes in some of the base model's own training data so the adapter keeps the base's general skills. The scored checkpoint is the one at the end of the epoch.

Data and provenance

Sources:

Which models wrote and checked the data. Almost all messages were written by Qwen3.8-Max (hosted). Urgency and sentiment were also rated by DeepSeek-V4-Flash (hosted), and labels were checked by Qwen3.8-Flash (hosted) and Qwen3.8-Flash-Next (local). Qwen3.8-Max is a hosted, closed-weight model, used through Alibaba Cloud's DashScope API to finish in time. Counts are on the 66,532 training rows; one row can pass through several models, so the counts do not add up to the total.

Urgency and sentiment targets are the mean of the generation plan and the two model ratings.

Training data not published. The test set is included in this repo as test.jsonl, and the calibration set (1,208 rows, the rows the threshold for the comparison with Qwen3.8-27B is chosen on) as calibration.jsonl.

Test set: test.jsonl, 7,256 rows: Messages to the 10% of the roughly 300 generated organisations that were never trained on, scored on all four questions (route, urgency, sentiment and whether a person is needed). It is byte for byte the set the adapter was scored on.

The terms of the hosted model providers are being checked for training and publication use.

Quality review. An independent reviewer checked the training, calibration and test files for shortcuts, duplicates, leaks between splits and junk before training. The report is at jeffhub.ai/adapters/triage/qa-report.

Limitations

• English only.
• Tied to the Jeff-Qwen3.5-0.8B v1.2 base. It will not work on any other base or version; the server refuses it if the base weights' checksum does not match. A v1.3 long-term-support base is coming, and the adapters will be retrained on it.
• Jeff chooses between the options you give it. It does not write text or reason in several steps.
• Calibration was fitted on this adapter's own calibration rows. On very different data, check it again.
• Everything listed under When not to use it above.

Links

• Adapter page: jeffhub.ai/adapters/triage
• Code and server: github.com/firelex/jeff
• Base model: mstrasser/Jeff-Qwen3.5-0.8B (revision v1.2)

Jeff is an independent project. It uses the same request format as Jev but is not affiliated with or endorsed by TypeSafe, the makers of Jev.