Ticket Intent PT-BR

Classifies messages sent to a Brazilian internet provider's support team into twelve intents. Fine-tuned from BERTimbau on synthetic data.

Trained only on synthetic data. It has never been evaluated on real customer messages.

Intended use

A portfolio and learning project. It could serve as a starting point for routing support messages, always with a confidence threshold and a person handling the rest. It should not make decisions about customers on its own.

Labels

sem_conexao, internet_lenta, conexao_instavel, wifi_senha, fatura_cobranca, mudanca_plano, mudanca_endereco, cancelamento, agendamento_visita, sem_relacao, saudacao_agradecimento, falar_atendente

Usage

from transformers import pipeline

classifier = pipeline("text-classification", model="GHSoliv/ticket-intent-ptbr")
classifier("a internet cai e volta o dia todo")

Results below 0.9382 confidence should go to a person.

Training data

8,560 messages generated with five Gemini models, split by generation batch into 7,848 train, 355 validation and 357 test messages. Five intents have about 1,300 training messages and the other seven have between 138 and 240. A separate set of 144 messages was drafted with an AI assistant, independently of the generator, and used only for reporting. The author reviewed all of them. Its messages are shorter than the training data, with a median of 7 words against 15.

Training

• Base model: neuralmind/bert-base-portuguese-cased
• Learning rate: 2e-5, with 10% warmup
• Batch size: 16
• Epochs: up to 6, with early stopping on validation macro-F1
• Max length: 128 tokens
• Seed: 44, chosen among 3 by validation score

Evaluation

Macro-F1, with 95% bootstrap intervals in parentheses:

This model beats the baseline on the test set (McNemar p < 0.001) but not on the independent test (p = 1.00). The test set shares the training data's writing style, so the independent result is the more cautious estimate.

With a confidence threshold of 0.9382, chosen on validation:

The model is confident even when it is wrong, so the threshold filters out few errors. Accuracy on the independent test falls short of the 97.5% target used on validation.

Browser version

onnx/model_quantized.onnx is a quantized copy for Transformers.js, used by the demo. Its macro-F1 is 0.969 on the test set and 0.900 on the independent test. The threshold was not recalibrated for it.

Limitations

• Performance on real support traffic is unknown.
• It assigns a single intent, even when a message contains two.
• sem_relacao is the weakest intent (F1 0.74 on the independent test), because unrelated topics cannot all be covered by one class.
• It confuses slow connections with unstable ones, and misses cancellations that are only implied.
• Synthetic data carries the habits of the models that wrote it.