Superseded: this model is from an earlier Jeff release and is no longer updated. The current release is Jeff v1.3: mstrasser/jeff-adapter-ground. This v1.2 adapter loads only on the v1.2 base.

Jeff-Qwen3.5-0.8B-ground

Passage re-ranking and answer grounding: picks the passage that answers a question, and checks whether an answer is supported by its sources.

Two jobs for retrieval-augmented generation, as two request shapes. Re-ranking: give it a question and 5 to 40 passages, and it picks the passage that answers it, or none. Grounding: give it source passages and an answer, and it says whether the answer is supported, partly supported, contradicted or unsupported, judged only from the sources. It is a LoRA adapter for Jeff-Qwen3.5-0.8B v1.2, a small open decision model: it answers in one forward pass, with no generated text to parse.

Results

Test set: Ground held-out documents, both questions, 4,160 rows. Requests from the 10% of source documents and articles that were never trained on, for both re-ranking and grounding. It is included in this repo as test.jsonl.

Calibration error is the expected calibration error over 15 confidence bins, after each model's fitted temperature: the average gap between how confident the model is and how often it is right. Lower is better. The untrained Qwen3.5-0.8B is read out from its own scores for the option letters.

Other test sets (planned, from public data): SQuAD 2.0 dev, re-ranking; HotpotQA dev (distractor), re-ranking; FEVER shared-task dev, grounding. Not measured yet.

Against Qwen3.8-27B

Here Jeff + ground answers first. Below a confidence threshold, the query is also sent to Qwen3.8-27B and the 27B's answer is used. Each adapter's threshold is the fastest one that still beats the 27B alone by at least 1 point of accuracy on the adapter's calibration rows (not the test rows).

On this sample ground's accuracy is about equal to the 27B's (96.3% against 96.7%, a difference within noise on 300 rows), at 20.1× the speed. It does not beat the 27B here. At threshold 0.64, 1.7% of queries were passed on to the 27B.

How this was measured: a fixed random sample of 300 held-out test rows, the same rows for both routes, on an Apple M4 Max (128 GB) with both models in MLX, one at a time. Qwen3.8-27B ran with 8-bit weights, prompted, with step-by-step reasoning off. Times are the mean per query over the whole sample, from prompt to answer. This is a 300-row sample, not the full test set.

Speed and memory

One request at a time on an NVIDIA RTX PRO 6000: 675 requests per setting, 75 test prompts of each of the nine adapters, interleaved. Measured for the adapters as a group, not for each adapter, so treat these as typical figures:

Jeff's speed and memory on the Mac: not measured yet.

Where it goes wrong

On the full test set:

• The commonest mistakes are in re-ranking, between a passage and none: picking none when a passage does answer, and picking a passage when none does. In grounding, contradicted is the weakest label, most often read as partly_supported.

Commonest confusions (right answer → chosen instead, rows):

Accuracy by right answer:

When to use it

• You run retrieval-augmented generation and want to pick the best of 5 to 40 retrieved passages, or learn that none answers the question.
• You want to check a generated answer against its sources before showing it, and tell apart supported, partly supported, contradicted and unsupported answers.
• You want both checks from one small model, fast enough to sit on every request.

When not to use it

• You need the answer written or the unsupported claim pointed out. Jeff only chooses between options; it does not generate text.
• Your passages are very long or very many. Training passages were 30 to 300 words, and every request stayed under 8,192 tokens.
• You need a judgement from outside knowledge. The grounding check is trained to judge only from the sources given.
• Your texts are mostly not in English. The training data is English.

How to use it

The adapter runs with Jeff's server on the Jeff-Qwen3.5-0.8B v1.2 base. One server holds the base and any number of adapters; each request picks one by name with "model": "ground".

git clone https://github.com/firelex/jeff && cd jeff
uv sync --no-default-groups --extra lora          # add --extra cuda on NVIDIA GPUs, --extra mac on Apple silicon
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B --revision v1.2 --local-dir Jeff-Qwen3.5-0.8B-v1.2
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B-ground --local-dir adapters/ground
JEFF_CHECKPOINT=Jeff-Qwen3.5-0.8B-v1.2 JEFF_ADAPTERS=adapters PORT=8765 \
  uv run --no-default-groups --extra lora jeff-serve
# add JEFF_BACKEND=mlx and --extra mac for Apple silicon

Every folder in adapters/ is served under its folder name. Add or replace an adapter while the server runs with POST /v1/adapters/reload. The server checks the base weights' checksum, so the adapter loads only on the exact v1.2 base.

from jeff import Client

ground = Client("http://localhost:8765", model="jeff-latest").with_model("ground")

Request format

State (the situation):

Questions:

• grounding (choice): Whether the answer is supported by the sources, judged only from the sources. Options: Four fixed options: supported, partly_supported, contradicted and unsupported, each with the one-line meaning shown in the example.

Rules:

• Use the instructions below word for word; the adapter was trained mostly on them. Grounding instructions, "Is the answer supported by the sources? Judge only from the sources, not from outside knowledge."
• Keep the four grounding option keys and their texts exactly as in the example. Their order does not matter; it was shuffled in training.
• The adapter also re-ranks passages, with a second request shape. State: one key, question, holding the user's question. Options: none (text "None of these passages answers the question") plus the passages as p1, p2, … with the passage text as the option text, optionally starting with its title in square brackets. Instructions, word for word: "Which passage answers the question? If none of them does, choose none."
• Re-ranking was trained with 5 to 40 passages of 30 to 300 words each. The passages change on every request, so nothing can be prepared in advance for them.
• Send one question per request; the two shapes have different state keys and are separate requests.

Example

{
  "model": "ground",
  "state": {
    "sources": "[1] Harbour Bridge\nThe Harbour Bridge opened to traffic in March 1932 after eight years of work. It carries eight lanes of road traffic and two railway lines.\n\n[2] Harbour ferries\nFerries have crossed the harbour since the 1840s. The busiest route runs from the Quay to Manly.",
    "answer": "The Harbour Bridge opened in 1934 and carries two railway lines."
  },
  "questions": {
    "grounding": {
      "type": "choice",
      "instructions": "Is the answer supported by the sources? Judge only from the sources, not from outside knowledge.",
      "criteria": {
        "supported": "Everything the answer claims is stated in or follows directly from the sources",
        "partly_supported": "Some claims are supported, but at least one is not in the sources",
        "contradicted": "The sources state something that conflicts with the answer",
        "unsupported": "The answer's main claim is not in the sources at all"
      }
    }
  }
}

curl -s localhost:8765/v1/systemone -H 'content-type: application/json' -d @request.json

The answer holds a probability for each option of each question, the chosen option and a confidence. (A recorded response to this example is not published yet.)

Training

The replay slice mixes in some of the base model's own training data so the adapter keeps the base's general skills. The scored checkpoint is the one at the end of the epoch.

Data and provenance

Sources:

Which models wrote and checked the data. Generated text was written by Qwen3.8-Max (hosted) and Qwen3.8-Flash-Next (local). Labels were checked by Qwen3.8-Flash (hosted), with second opinions from DeepSeek-V4-Flash (hosted). Qwen3.8-Max is a hosted, closed-weight model, used through Alibaba Cloud's DashScope API to finish in time. Counts are on the 50,760 training rows; one row can pass through several models, so the counts do not add up to the total.

Rows built from SQuAD 2.0, HotpotQA and FEVER keep their public text; the jobs above show which models wrote the generated text and checked the rows.

Training data not published. The test set is included in this repo as test.jsonl, and the calibration set (1,440 rows, the rows the threshold for the comparison with Qwen3.8-27B is chosen on) as calibration.jsonl.

Test set: test.jsonl, 4,160 rows: Requests from the 10% of source documents and articles that were never trained on, for both re-ranking and grounding. It is the exact set the adapter was scored on, except that in 760 rows a machine name in the source field now reads "local". Nothing else was changed.

The terms of the hosted model providers are being checked for training and publication use.

Quality review. An independent reviewer checked the training, calibration and test files for shortcuts, duplicates, leaks between splits and junk before training. The report is at jeffhub.ai/adapters/ground/qa-report.

Limitations

• English only.
• Tied to the Jeff-Qwen3.5-0.8B v1.2 base. It will not work on any other base or version; the server refuses it if the base weights' checksum does not match. A v1.3 long-term-support base is coming, and the adapters will be retrained on it.
• Jeff chooses between the options you give it. It does not write text or reason in several steps.
• Calibration was fitted on this adapter's own calibration rows. On very different data, check it again.
• Everything listed under When not to use it above.

Links

• Adapter page: jeffhub.ai/adapters/ground
• Code and server: github.com/firelex/jeff
• Base model: mstrasser/Jeff-Qwen3.5-0.8B (revision v1.2)

Jeff is an independent project. It uses the same request format as Jev but is not affiliated with or endorsed by TypeSafe, the makers of Jev.