Survey Feedback Actionability Classifier

A fine-tuned DistilBERT model that classifies open-ended survey feedback as either actionable or non-actionable.

The model is designed as an experimental business analytics tool to help organizations identify suggestions, recommendations, and opportunities for improvement within qualitative survey responses.

Model description

This model is a fine-tuned version of DistilBERT, trained to identify whether a survey response contains actionable feedback.

It uses natural language processing (NLP) to classify text into two categories:

Label	Description
`actionable`	Contains a specific suggestion, recommendation, request, or identifiable opportunity for improvement.
`non_actionable`	Expresses a general opinion, positive feedback, or personal experience without suggesting a specific improvement.

Examples:

Survey response	Classification
The registration process should be simplified.	Actionable
I would prefer more time for networking.	Actionable
The organization should offer virtual attendance options.	Actionable
The speakers were knowledgeable and engaging.	Non-actionable
I really enjoyed attending the conference.	Non-actionable

The model was developed as a practical demonstration of how AI can support the analysis of qualitative survey data.

Intended uses

The model is intended for experimental and educational use, particularly in business analytics, survey analysis, and feedback management.

Potential applications include:

Survey analysis: Identify responses that contain suggestions or opportunities for improvement.
Feedback triage: Help organize large collections of open-ended survey responses for further review.
Business intelligence: Support the process of turning qualitative feedback into information that can inform organizational decisions.
Portfolio and learning projects: Demonstrate applied NLP and text classification using Hugging Face Transformers.

The model can be used to classify individual responses or as part of a batch analysis workflow.

Limitations

This model has several important limitations:

Small synthetic dataset: The model was trained and evaluated using a small dataset of 100 synthetic survey responses. These examples do not represent the full range of language, writing styles, and feedback found in real-world surveys.
Limited generalizability: Performance on synthetic evaluation data does not establish how accurately the model will classify actual survey responses.
Context sensitivity: Some responses may require additional context to determine whether they represent an actionable suggestion.
Subjective classification: The distinction between actionable and non-actionable feedback can depend on organizational goals and interpretation.
Human review required: Predictions should be treated as preliminary classifications, not final decisions. Human review is recommended before using results to guide organizational improvements.

Confidence scores represent the model's prediction scores and should not be interpreted as a guarantee of correctness.

This model is an experimental proof of concept and is not intended to replace human judgment or serve as a production-ready decision-making system.

Training and evaluation data

The model was trained using a synthetic dataset of 100 survey feedback responses created for this project.

The examples were designed to represent open-ended feedback commonly encountered in professional development events, conferences, higher education, and nonprofit settings.

The dataset contains two labels:

0: non_actionable
1: actionable

The dataset was divided into training and evaluation subsets:

Split	Records	Percentage
Training	80	80%
Evaluation	20	20%
Total	100	100%

The training and evaluation data were used to fine-tune and assess the model's ability to distinguish between the two feedback categories.

Dataset: (https://huggingface.co/datasets/mriamah/survey-feedback-actionability)

Training procedure

The model was fine-tuned using the Hugging Face Transformers library with the pretrained distilbert/distilbert-base-uncased architecture.

Training hyperparameters

Hyperparameter	Value
Base model	`distilbert/distilbert-base-uncased`
Learning rate	0.0001
Training batch size	32
Evaluation batch size	32
Random seed	42
Optimizer	AdamW (PyTorch fused)
Optimizer betas	(0.9, 0.999)
Optimizer epsilon	1e-08
Learning rate scheduler	Linear
Number of epochs	10

Training results

The following results were recorded during training and evaluation:

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.0429	1	3	0.0409	1.0
0.0171	2	6	0.0176	1.0
0.0080	3	9	0.0084	1.0
0.0045	4	12	0.0058	1.0
0.0031	5	15	0.0054	1.0
0.0022	6	18	0.0060	1.0
0.0018	7	21	0.0066	1.0
0.0016	8	24	0.0070	1.0
0.0015	9	27	0.0072	1.0
0.0015	10	30	0.0072	1.0

The model achieved an evaluation accuracy of 1.0 (100%) on the synthetic evaluation split. The lowest recorded validation loss was 0.0054.

These results reflect performance on the small synthetic dataset and should not be interpreted as expected performance on real-world survey data.

Framework versions

Transformers 5.22.1
PyTorch 2.14.1+cpu
Datasets 4.8.5
Tokenizers 0.23.3

Usage

The model can be loaded using the Hugging Face Transformers pipeline:

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="mriamah/hf_survey-feedback-actionability",
    top_k=None
)

result = classifier(
    "The registration process should be simplified."
)

print(result)

The model returns prediction scores for both classification labels.

Project context

This project explores how AI and NLP can help organizations make better use of qualitative survey feedback.

Rather than manually reviewing every response, an organization could use an initial classification to help organize feedback for further analysis. The goal is to support the process of identifying improvement opportunities while keeping human review part of the decision-making process.

Loading…