Arcee Blitz Caller Beta
This repository contains a beta version of the Arcee Blitz Caller, a powerful tool for automated function calling and tool selection. Below you'll find instructions on how to launch the service using vllm as well as detailed performance metrics.
Overview
Arcee Blitz Caller is built on top of the Mistral-Small-24B-Base-2501 model and is designed for efficient and accurate tool calling operations.
Base Model
Quick Start
To launch the service using vllm, run the following command:
vllm serve arcee-ai/Arcee-Blitz-Caller \
--enable-auto-tool-choice \
--tool-call-parser hermes
Performance Metrics
Non-Live Performance
| Metric | Score |
|---|---|
| AST Accuracy | 85.15% |
| Simple AST | 70.08% |
| Multiple AST | 93.50% |
| Parallel AST | 89.50% |
| Parallel Multiple AST | 87.50% |
Live Performance
| Metric | Score |
|---|---|
| Accuracy | 74.19% |
| Simple AST | 71.71% |
| Multiple AST | 75.02% |
| Parallel AST | 43.75% |
| Parallel Multiple AST | 66.67% |
Multi-Turn Performance
| Metric | Score |
|---|---|
| Accuracy | 0.25% |
| Base | 0.50% |
| Miss Function | 0.00% |
| Miss Parameter | 0.00% |
| Long Context | 0.50% |
| Relevance Detection | 61.11% |
| Irrelevance Detection | 77.68% |
Note: The model demonstrates strong performance in single-turn interactions but currently shows limitations in multi-turn scenarios. We are actively developing new training data to enhance multi-turn capabilities.
Current Status
This is a beta release and is under active development. While the model shows promising results in many areas, users should be aware of its current limitations, particularly in multi-turn interactions.
License
This project is licensed under the Apache-2.0 License - see the LICENSE file for details.