Prompt Authenticator (Intent to Talk to Named Person)

Overview

This model is a fine-tuned version of distilbert-base-uncased on the nikolina-p/intention-request-to-talk-to dataset.

It is a binary classifier that detects whether a user prompt expresses intent to engage in a live conversation with a specific named person.

It is designed as a guardrail component in a multi-stage authentication pipeline.

---

Task

• 1 (Positive): User intends to talk/speak/connect or get live help from a named individual
• 0 (Negative): No such intent (e.g., message relay, references to teams, offices, or unnamed individuals, indirect or future/async communicationinfo requests, non-person targets)

---

Intended Use

This model is not a final decision-maker, but a first-stage filter in an authentication pipeline. 👉 The classifier is intentionally permissive to avoid false negatives
👉 Final authorization is handled downstream

---

Limitations

• Synthetic data may not reflect real-world distribution
• Struggles with implicit intent and long prompts
• Mild overconfidence on some edge cases
• English only

---

Training

• Base: distilbert-base-uncased
• Task: Binary classification
• Training: Hugging Face Trainer
• Early stopping enabled (best model ~epoch 4–5)

---

Training and evaluation data

The dataset is synthetically generated and iteratively refined via error analysis nikolina-p/intention-request-to-talk-to

Covers:

• direct and indirect requests
• imperative phrasing
• assist/help vs message/relay
• multi-sentence and adversarial cases

---

Evaluation results

The best model corresponds to the checkpoint from epoch 4 and achieves the following results on the evaluation set:

---

Training procedure

The classifier is intentionally somewhat permissive to reduce false negatives on genuine request-to-speak prompts. This was achieved mainly through labeling policy and iterative dataset refinement: borderline valid requests were labeled as positive, and common false-negative patterns were added back into training data. Early stopping used F1 on the positive class to keep a reasonable balance between recall and precision.

The best model is selected from checkpoints using:

• highest F1_1 (primary criterion)
• lowest evaluation loss (secondary criterion in case of ties or plateau)

---

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 8

---

Training results

---

Framework versions

• Transformers 5.2.0
• Pytorch 2.10.0+cu128
• Datasets 4.5.0
• Tokenizers 0.22.2