ruT5 smart home function calling
Fine-tuned ai-forever/ruT5-base. Takes a Russian smart home command and returns a JSON list of function calls in Home Assistant style (HassTurnOn, HassTurnOff, HassLightSet, HassClimateSetTemperature and others).
Модель превращает русскую команду для умного дома в JSON со списком вызовов функций.
Usage
The tokenizer vocabulary lacks some characters ({ } Q Y Z & é →), so the model works with an escaped text format. inference.py in this repository handles it. Do not call generate directly without it.
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("ilyali034/ruT5-base-smart-home")
sys.path.insert(0, path)
from inference import SmartHomeModel
model = SmartHomeModel(path)
print(model.predict("включи свет на кухне"))
Output example:
{"text": "[{\"name\": \"HassTurnOn\", \"arguments\": {\"name\": \"Kitchen Light\"}}]", "calls": [{"name": "HassTurnOn", "arguments": {"name": "Kitchen Light"}}]}
Training
- Base model: ai-forever/ruT5-base
- Data: RockMan256/needle-ru-opt, split 90/5/5 (train/val/test), seed 42
- 5 epochs, Adafactor, learning rate 3e-4, batch size 16, fp16
- The tools list is identical in every example of the dataset, so it is not given to the model: the input is
query: <command>
Results (test, 1290 examples)
| Metric | Value |
|---|---|
| Exact match (canonical JSON) | 0.828 |
| Valid JSON rate | 1.000 |
| Function name F1 (micro) | 0.993 |
| Main function accuracy | 0.997 |
| Arguments accuracy | 0.792 |
| Token F1 | 0.956 |
| BLEU | 97.3 |
| chrF | 96.9 |
Limitations
- Almost all errors are in the
nameargument (device name). The model does not see the list of devices in the home, so when a command does not say which device is meant (for example, several thermostats), or a device has several valid English names, it picks one. Use it together with a list of real devices, or map its output to the closest known device name. - Sometimes the model skips the last call in commands with several actions.
- Device names in the data are mostly English, commands are Russian.
- Trained on synthetic data. Quality on real user phrases may be lower.