Model Details
This model is designed to classify customer service inquiries into two requst type: Inicident and Service Request.
Training Data
The model was trained on a balanced dataset of 28000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance.
https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt
Training Procedure
The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.
Model Performance
The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:
Confusion Matrix Analysis
- Incident: The model has a high number of correct classifications with 2500 true positives. There are 25 false negatives where "Incident" was incorrectly classified as "Service Request".
- Service Request: There are 440 true positives where "Service Request" was correctly classified. However, there are 150 false negatives where "Service Request" was incorrectly classified as "Incident".
Learning Curve Analysis
- Training Loss: The training loss starts at approximately 0.20 and shows a steady decrease, ending near 0.12. This indicates that the model is effectively learning and improving its performance on the training dataset.
- Validation Loss: The validation loss begins around 0.18 and, after a slight decrease, starts to plateau around 0.16. This suggests that while the model is learning and generalizing, improvements on the validation set are limited after the initial epochs.
Interprating Model's Output:
- LABEL_0 stands for Incident
- LABEL_1 stands for Service Request
