OVOS intent classifier, multilingual

A static-embedding intent classifier for OpenVoiceOS. It maps an utterance to one of 221 intent labels of the form <skill_id>:<intent_name>, the names the OVOS m2v pipeline registers at runtime. It is the default model of ovos-m2v-pipeline and the fallback for every language without a model of its own. English also has a smaller English-only model, ovos-m2v-intents-en.

Inference is a vector lookup, an average and one linear layer, so it runs on a CPU. The embedding table is 500,353 tokens by 256 dimensions in float32, about 530 MB on disk and in memory.

Use

The OVOS pipeline loads this repository by default. To load it directly:

from model2vec.inference import StaticModelPipeline

model = StaticModelPipeline.from_pretrained("OpenVoiceOS/ovos-m2v-intents-multilingual")
model.predict(["set a timer for ten minutes"])

labels.json lists the 221 labels the model was trained on. The pipeline reads it and matches only those labels.

Smaller versions and ONNX

The repository root holds the float32 model, which the OVOS pipeline loads by default. Three more copies sit in folders beside it. In fp16 and int8 only the embedding table changes precision; the labels and the classifier layer are the same.

To use a smaller copy in OVOS, set subfolder:

{"intents": {"ovos-m2v-pipeline": {
  "model": "OpenVoiceOS/ovos-m2v-intents-multilingual",
  "subfolder": "int8"}}}

In Python, download one folder and load it:

from huggingface_hub import snapshot_download
from model2vec.inference import StaticModelPipeline

path = snapshot_download("OpenVoiceOS/ovos-m2v-intents-multilingual", allow_patterns=["int8/*"])
model = StaticModelPipeline.from_pretrained(f"{path}/int8")

onnx/model.onnx runs without model2vec. It takes input_ids and attention_mask (int64, batch by tokens) and returns probabilities (float32, batch by 221), in the order of head_config.classes in onnx/config.json. Tokenize with onnx/tokenizer.json without special tokens and keep at most 512 tokens:

import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("onnx/tokenizer.json")
classes = json.load(open("onnx/config.json"))["head_config"]["classes"]
ids = tok.encode("set a timer for ten minutes", add_special_tokens=False).ids[:512]
sess = ort.InferenceSession("onnx/model.onnx")
probs = sess.run(None, {"input_ids": np.array([ids]), "attention_mask": np.ones((1, len(ids)), np.int64)})[0]
print(classes[probs[0].argmax()])

One-language versions

For devices with little memory, such as a Raspberry Pi, a folder per language holds an int8 copy of this model with its vocabulary cut to that language. It keeps the tokens the language's training sentences use and the tokens of its 50,000 most common words (20,000 for Spanish; Galician and Basque keep only the tokens of their training sentences). The embeddings of the kept tokens, the labels and the classifier layer are unchanged; nothing is retrained. A word whose tokens were cut is not seen, so a sentence with rare words can score lower than the gold sentences show.

Gold rows are the language's rows of the gold set; the last column is the full float32 model on the same rows.

From ovos-m2v-pipeline 0.36.0a2, a models entry names the folder for its language:

{"intents": {"ovos-m2v-pipeline": {"models": {
  "pt": {"model": "OpenVoiceOS/ovos-m2v-intents-multilingual", "subfolder": "pt"}}}}}

Earlier models

The model trained on the same data in 128 shards taken in order, which scores 0.8155 on all gold rows, is the tag v7-preview2. Set "revision": "v7-preview2" to load it.

The model before it, with 210 labels in the older label names, is revision 145a1bd94acb2620d9aa92c82567cf2bbb55f0a2. In the OVOS configuration:

{"intents": {"ovos-m2v-pipeline": {
  "model": "OpenVoiceOS/ovos-m2v-intents-multilingual",
  "revision": "145a1bd94acb2620d9aa92c82567cf2bbb55f0a2"}}}

In Python:

from huggingface_hub import snapshot_download
from model2vec.inference import StaticModelPipeline

path = snapshot_download("OpenVoiceOS/ovos-m2v-intents-multilingual",
                         revision="145a1bd94acb2620d9aa92c82567cf2bbb55f0a2")
model = StaticModelPipeline.from_pretrained(path)

Training

The held-out side has 14,969 templates. 3,004 of their rows repeat a training utterance and are left out of the score, which leaves 457,445 rows. Training used the other 135,695 templates, 4,491,557 rows.

Results

The gold sentences are the skills' own golden utterances, not generated from the templates the model trains on. 10,496 of the 12,908 are flagged needs_manual; they are scored like the others. The last two rows compare this model and the 210-label model on the gold rows whose label both can emit, 85 labels.

Gold accuracy per language:

Languages with fewer than 10 gold rows are not meaningful and are left out of the table: bg-BG (9), he-IL (9), hr-HR (9), id-ID (9), ja-JP (9), nb-NO (9), sk-SK (9), sl-SI (9), th-TH (9), ar-XA (6), et-EE (6), fil-PH (6), ko-KR (6), lv-LV (6), ms-MY (6), uk-UA (6), vi-VN (6), zh-CN (6), ar-XX (3), cmn-CN (3), is-IS (3), nn-NO (3), zsm-MY (3).

Limits

• Labels are skewed. Templates that multiply slot values produce most of the rows. ovos-skill-date-time:what_time_will_it_be is 18.9% of the corpus and ovos-skill-pokepedia:battle 15.0%; in Kabyle the first is 78.5% of the rows and in Polish 69.4%. The median label has 479 rows and the smallest 10.
• Place names are not declined. A {location} slot is filled with the name as written, so a Polish row reads "we Irlandia" where a speaker says "w Irlandii". The model has seen few correctly inflected place names in languages that decline them.
• The gold sentences are model-written. Every gold sentence in the fleet was written by a language model. needs_manual marks rows meant for a human check, and whether that check happened is not recorded, so the gold figures measure agreement with model-written sentences, not with recorded users.

License

Apache-2.0. The base model is MIT.