Funcdex-0.6B-gmail

Funcdex-0.6B-gmail is a research preview model by Prem Labs. It has been trained on a mix of Funcdex-MT-Function-Calling, Instruct-Following, Single-turn function datasets. It is a LoRA finetune of Qwen3-0.6B (with thinking disabled).

This model excels at Multi-turn Function Calling with tools from gmail.

The code used to generate the dataset can be found here.

Evaluation

Results

BFCL v3

• We filtered BFCLv3 examples relevant to the toolkits/bundles and report performance:
• The filtered set is only 83 examples. Further emphasizing the need for workflow/toolkit-specialized workflows.

Funcdex-MT: Overall Performance

Funcdex-MT: Toolkit-Level Performance

• Funcdex-0.6B are specialized models. Reported number is the average performance of each specific model in their respective subset.

Funcdex-MT: Bundle/Multi-toolkit Performance:

Deployment with vLLM

vllm serve ojus1/Qwen3-0.6B-Instruct --enable-lora --lora-modules prem-research/Funcdex-0.6B=prem-research/Funcdex-0.6B-gmail --enable-auto-tool-choice --tool-call-parser hermes

Inference

• Given a conversation, we extract all tuples (context_messages, function_calls) and use it to generate predictions. We ignore the content field and only evaluate function_calls generated by an LLM.
• We use vLLM deployment with tool_choice="auto".

Metrics

Given a list of predicted and reference function calls, we report two metrics:

• Function Call String Match (SR): We perform greedy match and report best-matched string ratio using difflib.SequenceMatcher.ratio. The number reported is average string ratio.
• Exact Match (EM): Same as above, but we perform exact string match instead. The number reported is EM F1 Score.

EM is a strict metric, and penalizes string arguments in function calls that may be "okay", e.g. "email_content": "This is an example." v/s "email_content": "This is an Example.", both only differ by one letter.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
import json

# Load model and tokenizer
base_model_name = "ojus1/Qwen3-0.6B-Instruct"
model_name = "prem-research/Funcdex-0.6B-gmail"

tokenizer = AutoTokenizer.from_pretrained(model_name)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype="auto",
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "SEND_EMAIL",
            "description": "Send an email message",
            "parameters": {
                "type": "object",
                "properties": {
                    "to": {"type": "string", "description": "Recipient email address"},
                    "subject": {"type": "string", "description": "Email subject"},
                    "body": {"type": "string", "description": "Email body content"}
                },
                "required": ["to", "subject", "body"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "FETCH_EMAILS",
            "description": "Fetch emails from Gmail",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "Search query"},
                    "max_results": {"type": "integer", "description": "Maximum number of results"}
                },
                "required": ["query"]
            }
        }
    }
]

# Define conversation
messages = [
    {"role": "system", "content": "You are a helpful assistant that can help with tasks by using tools."},
    {"role": "user", "content": "Send an email to john@example.com with subject 'Meeting Update' and let him know the meeting is rescheduled to 3 PM tomorrow."}
]

# Apply chat template with tools
formatted_input = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True
)

# Tokenize and generate
input_tokens = tokenizer(formatted_input, return_tensors="pt").to(model.device)
output = model.generate(**input_tokens, max_new_tokens=256, do_sample=False)
response = tokenizer.decode(output[0][input_tokens['input_ids'].shape[1]:], skip_special_tokens=True)

print("Response:", response)

For best results, provide detailed system-prompt to steer the tool-use behaviour.

License

The models, code and the dataset are licensed under MIT License.