Superseded: this model is from an earlier Jeff release and is no longer updated. The current release is Jeff v1.3: mstrasser/jeff-adapter-support-intents. This v1.2 adapter loads only on the v1.2 base.

Jeff-Qwen3.5-0.8B-support-intents

Customer request intents: names what a customer or assistant user is asking for, from a fixed list of requests such as cancel_order or track_refund.

Give it a short customer or assistant message and a list of requests with one-line descriptions, and it returns a probability for each request. It was trained on three fixed lists: 27 online-shop requests (Bitext), 64 home-assistant requests (HWU64) and 7 voice-assistant requests (SNIPS). It is a LoRA adapter for Jeff-Qwen3.5-0.8B v1.2, a small open decision model: it answers in one forward pass, with no generated text to parse.

Results

Test set: Bitext, HWU64 and SNIPS hold-out, 5,577 rows. 10% of the Bitext and HWU64 messages, held out by a stable hash of the text, plus the official SNIPS June 2017 held-out files; never trained on. It is included in this repo as test.jsonl.

Model	Accuracy	Calibration error (ECE)
Qwen3.5-0.8B, untrained	33.9%	0.162
Jeff v1.2 0.8B alone	85.1%	0.080
Jeff v1.2 0.8B + support-intents	96.8%	0.006

Calibration error is the expected calibration error over 15 confidence bins, after each model's fitted temperature: the average gap between how confident the model is and how often it is right. Lower is better. The untrained Qwen3.5-0.8B is read out from its own scores for the option letters.

Against Qwen3.8-27B

Here Jeff + support-intents answers first. Below a confidence threshold, the query is also sent to Qwen3.8-27B and the 27B's answer is used. Each adapter's threshold is the fastest one that still beats the 27B alone by at least 1 point of accuracy on the adapter's calibration rows (not the test rows).

Route	Accuracy	Mean time per query	Sent to the 27B
Qwen3.8-27B alone	86.0%	6.44 s	100%
Jeff + support-intents, 27B below threshold 0.00	95.3%	118 ms	0.0%

The threshold is 0.00: on the calibration rows the adapter alone already beat the 27B by more than a point, so no query is passed on. The route is 54.6× faster than the 27B alone.

How this was measured: a fixed random sample of 300 held-out test rows, the same rows for both routes, on an Apple M4 Max (128 GB) with both models in MLX, one at a time. Qwen3.8-27B ran with 8-bit weights, prompted, with step-by-step reasoning off. Times are the mean per query over the whole sample, from prompt to answer. This is a 300-row sample, not the full test set.

Speed and memory

One request at a time on an NVIDIA RTX PRO 6000: 675 requests per setting, 75 test prompts of each of the nine adapters, interleaved. Measured for the adapters as a group, not for each adapter, so treat these as typical figures:

Setting	Median	95th percentile	GPU memory after loading
The v1.2 base alone	25.9 ms	35.5 ms	1.74 GB
Shared base, one adapter (default mode)	31.2 ms	40.3 ms	1.79 GB
Shared base, all nine adapters, a different one on every request	30.0 ms	39.4 ms	1.96 GB
Merged mode (`JEFF_ADAPTER_MODE=merged`), one adapter folded into the weights	25.7 ms	35.7 ms	1.77 GB

Jeff's speed and memory on the Mac: not measured yet.

Where it goes wrong

On the full test set:

Errors cluster in the home-assistant (HWU64) requests, which overlap in meaning: general_quirky against qa_factoid and news_query, play_radio against play_music, calendar_set against calendar_query. Online-shop (Bitext) requests are almost always right, but Bitext's test messages are template-made and close to its training messages, so that score is easier than real traffic.

Commonest confusions (right answer → chosen instead, rows):

Right answer	Chosen instead	Rows
`general_quirky`	`news_query`	7
`qa_factoid`	`general_quirky`	5
`general_quirky`	`qa_factoid`	5
`calendar_query`	`general_quirky`	5
`search_screening_event`	`search_creative_work`	5
`takeaway_order`	`takeaway_query`	4

Weakest answers (answers with at least 20 test rows; 97 answers in all):

Right answer	Test rows	Accuracy
`takeaway_order`	20	70.0%
`general_quirky`	115	70.4%
`music_query`	30	80.0%
`recommendation_events`	31	83.9%
`calendar_query`	84	85.7%
`play_radio`	49	85.7%

By source data set (accuracy / calibration error):

Group	Test rows	Qwen3.5-0.8B untrained	Jeff alone	Jeff + adapter
Bitext (online shop)	2,361	43.6% / 0.260	84.0% / 0.082	99.9% / 0.001
HWU64, message text also in the base model's training data	897	19.5% / 0.084	88.5% / 0.069	92.6% / 0.017
HWU64, message text not in the base model's training data	1,624	17.5% / 0.082	83.5% / 0.103	93.6% / 0.013
SNIPS (voice assistant)	695	57.6% / 0.222	87.9% / 0.042	98.8% / 0.007

HWU64 is the text source of MASSIVE, which is in the base model's training data, so some HWU64 test messages were seen by the base (never by the adapter). The unseen HWU64 rows are the fairest test.

When to use it

You run an online shop chat or a voice assistant and want each message matched to one request from a fixed list.
Your requests are close to the ones in training (27 online-shop requests, 64 home-assistant requests, 7 voice-assistant requests), each described in one line.
Messages are short and in English.

When not to use it

Your list of requests is very different from the training lists. Try it, but measure first; the triage adapter reads your own team descriptions.
A message holds several requests and you need all of them. The question picks one.
You need the reply written. Jeff chooses between options; it does not generate text.
Your messages are long emails or tickets. Training messages are short single requests.

How to use it

The adapter runs with Jeff's server on the Jeff-Qwen3.5-0.8B v1.2 base. One server holds the base and any number of adapters; each request picks one by name with "model": "support-intents".

git clone https://github.com/firelex/jeff && cd jeff
uv sync --no-default-groups --extra lora          # add --extra cuda on NVIDIA GPUs, --extra mac on Apple silicon
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B --revision v1.2 --local-dir Jeff-Qwen3.5-0.8B-v1.2
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B-support-intents --local-dir adapters/support-intents
JEFF_CHECKPOINT=Jeff-Qwen3.5-0.8B-v1.2 JEFF_ADAPTERS=adapters PORT=8765 \
  uv run --no-default-groups --extra lora jeff-serve
# add JEFF_BACKEND=mlx and --extra mac for Apple silicon

Every folder in adapters/ is served under its folder name. Add or replace an adapter while the server runs with POST /v1/adapters/reload. The server checks the base weights' checksum, so the adapter loads only on the exact v1.2 base.

from jeff import Client

support_intents = Client("http://localhost:8765", model="jeff-latest").with_model("support-intents")

Request format

State (the situation):

Key	Changes per request	What it holds
`service`	no	One short phrase about the service the message is sent to. Training used three, one per data set, for example "Customer support chat of an online shop".
`message`	yes	The customer's or user's message as received.

Questions:

intent (choice): What the customer wants, as the request that best matches their message. Options: In training, each message's options were all the requests of its own data set: 27 for the online shop (Bitext), 64 for the home assistant (HWU64) and 7 for the voice assistant (SNIPS). Keys are snake_case names such as track_order, each with a one-line description. The full lists are BITEXT, HWU64 and SNIPS in descriptions.py in the source.

Rules:

Use the option keys and descriptions from descriptions.py where they fit your service; the adapter was trained on them.
Use the instructions below word for word; the adapter was trained mostly on them.

Example

{
  "model": "support-intents",
  "state": {
    "service": "Customer support chat of an online shop",
    "message": "hi, I sent back the jacket two weeks ago and still haven't seen the money. where is it?"
  },
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "What does the customer want? Choose the request that best matches what the customer is asking for in their message.",
      "criteria": {
        "track_refund": "Check the status of a refund they are expecting.",
        "get_refund": "Get their money back for a purchase.",
        "check_refund_policy": "Learn the refund policy and whether they qualify for a refund.",
        "track_order": "Find out where their order is or its current status.",
        "cancel_order": "Cancel an order they placed.",
        "change_order": "Change an existing order (for example add, remove or swap items).",
        "payment_issue": "Report or solve a problem with a payment.",
        "complaint": "Make a complaint about the product, service or company.",
        "contact_human_agent": "Talk to a human agent instead of an automated assistant.",
        "delivery_period": "Find out when an order will arrive or how long delivery takes."
      }
    }
  }
}

curl -s localhost:8765/v1/systemone -H 'content-type: application/json' -d @request.json

The answer holds a probability for each option of each question, the chosen option and a confidence. (A recorded response to this example is not published yet.)

Training


Base	Jeff-Qwen3.5-0.8B v1.2 (final checkpoint of run `0.8b-20260929-2258`)
Method	LoRA, rank 16
Rows	60,334 (the adapter's rows plus a replay slice of the v1.2 base training mix, 10% of the adapter rows)
Epochs	1
Steps	943
Time	86.7 minutes on one NVIDIA RTX PRO 6000
Adapter files	41.5 MB
Run	`0.8b-support-intents-20260930-1209`, step 943

The replay slice mixes in some of the base model's own training data so the adapter keeps the base's general skills. The scored checkpoint is the one at the end of the epoch.

Data and provenance

Sources:

Source	Licence	Notes
Bitext customer support training data set (27 intents)	CDLA-Sharing-1.0	Only the customer message and intent columns are used; the assistant response column is never read. Under CDLA, models trained on the data are Results, not Data.
HWU64 (Liu et al. 2019, NLU-Evaluation-Data)	CC-BY-4.0	Home-assistant requests. Four very small intents outside the usual 64 are left out.
SNIPS custom intent engines benchmark, June 2017 (Coucke et al. 2018)	CC0-1.0	Seven voice-assistant intents.

Training data: built from the public sources above. A script to rebuild our rows from them will follow. The test set is included in this repo as test.jsonl.

Test set: test.jsonl, 5,577 rows: 10% of the Bitext and HWU64 messages, held out by a stable hash of the text, plus the official SNIPS June 2017 held-out files; never trained on.

Calibration set: calibration.jsonl, 999 rows: the rows the threshold for the comparison with Qwen3.8-27B is chosen on; never trained on. It is byte for byte the set the adapter was scored on.

Quality review. An independent reviewer checked the training, calibration and test files for shortcuts, duplicates, leaks between splits and junk before training. The report is at jeffhub.ai/adapters/support-intents/qa-report.

Limitations

English only.
Tied to the Jeff-Qwen3.5-0.8B v1.2 base. It will not work on any other base or version; the server refuses it if the base weights' checksum does not match. A v1.3 long-term-support base is coming, and the adapters will be retrained on it.
Jeff chooses between the options you give it. It does not write text or reason in several steps.
Calibration was fitted on this adapter's own calibration rows. On very different data, check it again.
Everything listed under When not to use it above.

jeff-legacy/Jeff-Qwen3.5-0.8B-support-intents