Doosra tool caller: Qwen3.5 4B fine-tuned to use a cricket analytics toolkit (v2)

Doosra is a cricket analytics app with a local AI copilot: you ask a question, and the model answers it by calling Doosra's 25 tools (leaderboards, player profiles, match-ups, venues, records, match replays, charts, SQL…) over 11 million balls of Cricsheet data, then writes an answer from the results.

This is Qwen3.5 4B fine-tuned on 6,695 of Doosra's own tool-calling conversations, so the small model does the job reliably, with a prompt about 8x shorter than the app gives general models (about 280 tokens instead of 2,100, plus compact tool schemas), which matters on a laptop where reading the prompt is the slow part. In Doosra's evaluation it answers 33 of 40 open questions right, against 30 for Qwen3.5 9B and 29 for the base 4B, about 2.5x faster (see Evaluation).

v2: rank 32, learning rate 1e-4, 2 epochs, dataset v2; the model Doosra ships.

What it does

Who led in dot % during the death overs of the BPL?

The base model abbreviates the competition and cuts the table to 5 rows nobody asked for. The fine-tuned model makes the exact call; Doosra runs it, and the answer is written from the result (this one is from the test set):

Rashid Khan is top with 49.07 dot % (Bangladesh Premier League, male, death).

• Nasum Ahmed (Khulna Tigers): 47.92
• Mohammad Irfan (Rajshahi Royals): 45.61

Files

Use

In Doosra (3.0 and later): Settings → AI model → Qwen3.5 4B Doosra. The app sends the short prompt and compact tools this model was trained with.

With llama.cpp (any OpenAI-compatible client). The model emits tool calls; your code runs the tools. Give it the system prompt and tool list it was trained with, from the dataset, with thinking off:

llama-server -hf Sarthak213/doosra-qwen3.5-4b-toolcalls-v2:Q4_K_M --jinja --ctx-size 16384

import json
from huggingface_hub import hf_hub_download
from openai import OpenAI

data = "Sarthak213/doosra-toolcalls"
tools = json.load(open(hf_hub_download(data, "tools.json", repo_type="dataset")))
system = open(hf_hub_download(data, "system_prompt.txt", repo_type="dataset"), encoding="utf-8").read()

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
reply = client.chat.completions.create(
    model="local", temperature=0, tools=tools,
    messages=[{"role": "system", "content": system},
              {"role": "user", "content": "Who led in dot % during the death overs of the BPL?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}})
print(reply.choices[0].message.tool_calls)

Send each tool's result back as a tool message and call again; the model finishes with a final_answer call.

Training

Data: Sarthak213/doosra-toolcalls, version 2: 8,000 conversations (6,695 train, 345 validation, 960 test) across 31 kinds of question, single calls and multi-step ones. Questions come from templates over real players, teams, venues and competitions, and 46% were reworded by Qwen3.5 9B; every tool call was executed against the real database, and every answer was written from the results. About 15% of players, teams and venues appear only in the test split, so the test measures names the model never saw. Version 2 adds single-figure questions, about 20 SQL patterns, recovery from a failed call, fixed answer templates, and keeps the end-to-end evaluation's players, grounds and facts out of training (see the dataset card).

Method: LoRA on unsloth/Qwen3.5-4B with Unsloth and TRL, on the language layers (attention and MLP; vision layers frozen), loss on the assistant's turns only (tool calls and answers, not the prompt, question or tool results). Conversations are rendered with Qwen's chat template, the tool list and thinking off, exactly as the app sends them.

Validation loss fell from 0.012 (step 100) to 0.0048 (step 500) and ended at 0.0050. Unlike v1, validation loss kept falling into the second epoch (v2's broader data had more to learn) and levelled off around step 500. The adapter at the end of epoch 1 was exported too (…-v2-epoch1); the two tie on the open questions, and this one is faster and more accurate on the auto-graded ones, so it's the one Doosra ships.

Evaluation

Measured with ToolEval, a local evaluation app, each model served by llama.cpp as Doosra runs it (Q4_K_M, thinking off): the base models with the app's full prompt and tool list, the fine-tuned ones with the short prompt they were trained on.

End to end: 40 open questions (Doosra's own evaluation set, written separately from the training templates and kept out of training), each answered through a real tool loop against Doosra's database. 35 are graded automatically on the facts the answer must contain (accepting name variants such as "V Kohli" for "Virat Kohli"); 5 are open-ended (a qualification or definition to choose) and were checked by hand against the database.

End to end (the same 40 questions for every model)

It answers more open questions right than both base models while making less than half the base 4B's calls and answering in under half the time. The four auto-graded questions it misses: a comparison that listed only the leader's average (q24), a COUNT(*) where COUNT(DISTINCT player) was needed (q34), and two questions where the definition differs from the reference (Kohli's matches: batting innings 808 against appearances 834; most wins: men's cricket by default against both genders).

Held-out test set (dataset v2: unseen players, teams and venues)

First call of 300 conversations, and every step of 100 with the final answer checked against the tool results.

Limitations

• Doosra's tools only. It learned these 25 tools, their arguments and Doosra's answer style. It isn't a general assistant or a general tool caller, and it expects the short system prompt it was trained with.
• Synthetic questions. Training questions come from templates, partly reworded; real users ask in more ways. The end-to-end questions are the check on that.
• SQL fine print. It writes valid SQL on Doosra's schema but can miss a rule in the schema notes: asked how many matches were ties, it counted winner IS NULL, which also includes draws and no-results. Check SQL answers to unusual questions.
• Names as the data has them. Cricsheet names players as "V Kohli", "JJ Bumrah"; the tools resolve most forms, but unusual spellings may miss.
• The data's coverage. Answers are only as complete as Cricsheet: some competitions and teams are thin or missing (for example, there are no Afghanistan matches).

Licence

The model is released under Apache 2.0, like Qwen3.5 4B. The training data's cricket results are from Cricsheet under the Open Data Commons Attribution License.