Doosra tool caller: Qwen3.5 4B fine-tuned to use a cricket analytics toolkit (v1, run A)

Superseded by v2. This first version is near-perfect on questions shaped like its training templates but answered fewer open questions right than the base model it started from. The evaluation below shows why, and v2 fixes it. Kept for the comparison.

Doosra is a cricket analytics app with a local AI copilot: you ask a question, and the model answers it by calling Doosra's 25 tools (leaderboards, player profiles, match-ups, venues, records, match replays, charts, SQL…) over 11 million balls of Cricsheet data, then writes an answer from the results.

This is Qwen3.5 4B fine-tuned on 5,854 of Doosra's own tool-calling conversations, so the small model does the job reliably, with a prompt about 8x shorter than the app gives general models (about 280 tokens instead of 2,100, plus compact tool schemas), which matters on a laptop where reading the prompt is the slow part. See Evaluation for how it compares with the base models and v2.

v1, run A: rank 16, learning rate 2e-4, 1 epoch, dataset v1.

What it does

Who led in dot % during the death overs of the BPL?

The base model abbreviates the competition and cuts the table to 5 rows nobody asked for. The fine-tuned model makes the exact call; Doosra runs it, and the answer is written from the result (this one is from the test set):

Rashid Khan is top with 49.07 dot % (Bangladesh Premier League, male, death).

• Nasum Ahmed (Khulna Tigers): 47.92
• Mohammad Irfan (Rajshahi Royals): 45.61

Files

Use

In Doosra (3.0 and later): Settings → AI model → Qwen3.5 4B Doosra. The app sends the short prompt and compact tools this model was trained with.

With llama.cpp (any OpenAI-compatible client). The model emits tool calls; your code runs the tools. Give it the system prompt and tool list it was trained with, from the dataset, with thinking off:

llama-server -hf Sarthak213/doosra-qwen3.5-4b-toolcalls-a:Q4_K_M --jinja --ctx-size 16384

import json
from huggingface_hub import hf_hub_download
from openai import OpenAI

data = "Sarthak213/doosra-toolcalls"
tools = json.load(open(hf_hub_download(data, "tools.json", repo_type="dataset")))
system = open(hf_hub_download(data, "system_prompt.txt", repo_type="dataset"), encoding="utf-8").read()

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
reply = client.chat.completions.create(
    model="local", temperature=0, tools=tools,
    messages=[{"role": "system", "content": system},
              {"role": "user", "content": "Who led in dot % during the death overs of the BPL?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}})
print(reply.choices[0].message.tool_calls)

Send each tool's result back as a tool message and call again; the model finishes with a final_answer call.

Training

Data: Sarthak213/doosra-toolcalls, version 1: 7,000 conversations (5,854 train, 306 validation, 840 test) across 29 kinds of question, single calls and multi-step ones. Questions come from templates over real players, teams, venues and competitions, and 43% were reworded by Qwen3.5 9B; every tool call was executed against the real database, and every answer was written from the results. About 15% of players, teams and venues appear only in the test split, so the test measures names the model never saw.

Method: LoRA on unsloth/Qwen3.5-4B with Unsloth and TRL, on the language layers (attention and MLP; vision layers frozen), loss on the assistant's turns only (tool calls and answers, not the prompt, question or tool results). Conversations are rendered with Qwen's chat template, the tool list and thinking off, exactly as the app sends them.

Validation loss fell from 0.029 (step 50) to 0.0057 (step 250) and ended at 0.0061.

Evaluation

Measured with ToolEval, a local evaluation app, each model served by llama.cpp as Doosra runs it (Q4_K_M, thinking off): the base models with the app's full prompt and tool list, the fine-tuned ones with the short prompt they were trained on.

End to end: 40 open questions (Doosra's own evaluation set, written separately from the training templates and kept out of training), each answered through a real tool loop against Doosra's database. 35 are graded automatically on the facts the answer must contain (accepting name variants such as "V Kohli" for "Virat Kohli"); 5 are open-ended (a qualification or definition to choose) and were checked by hand against the database.

End to end (the same 40 questions for every model)

v1 was near-perfect on its templated tests but lost end to end: it couldn't write SQL (the compact tool list had dropped the schema), answered with a template's headline figure instead of the one asked (sixes, balls faced), made one call and stopped where a second was needed, and had learned answers with no figures from a template bug. Dataset v2 fixed each of these.

Held-out test set (dataset v1: unseen players, teams and venues)

First call of 300 conversations, and every step of 100 with the final answer checked against the tool results.

Limitations

• Doosra's tools only. It learned these 25 tools, their arguments and Doosra's answer style. It isn't a general assistant or a general tool caller, and it expects the short system prompt it was trained with.
• Synthetic questions. Training questions come from templates, partly reworded; real users ask in more ways. The end-to-end questions are the check on that.
• SQL fine print. It writes valid SQL on Doosra's schema but can miss a rule in the schema notes: asked how many matches were ties, it counted winner IS NULL, which also includes draws and no-results. Check SQL answers to unusual questions.
• Names as the data has them. Cricsheet names players as "V Kohli", "JJ Bumrah"; the tools resolve most forms, but unusual spellings may miss.
• The data's coverage. Answers are only as complete as Cricsheet: some competitions and teams are thin or missing (for example, there are no Afghanistan matches).

Licence

The model is released under Apache 2.0, like Qwen3.5 4B. The training data's cricket results are from Cricsheet under the Open Data Commons Attribution License.