akifkulll/mayi-distilbert-b
distilbert-base-uncased fine-tuned to decide whether an AI agent's proposed tool call should be
allowed (0) or denied (1). Input format B: the tool call, then the user's request, then the agent's earlier steps (max 512 tokens; the end of the history is cut).
Trained for the COE025 NLP course project (Istinye University) on the cleaned English split of auto-1b-data (cleaning and splits).
Test results (threshold 0.5)
| rows | accuracy | macro-F1 | false allow rate | false deny rate | AUROC |
|---|---|---|---|---|---|
| 71613 | 0.9607 | 0.9607 | 0.0405 | 0.0381 | 0.9929 |
Use
from transformers import pipeline
clf = pipeline("text-classification", model="akifkulll/mayi-distilbert-b")
clf("### PROPOSED TOOL CALL\ntool: shell\nargs: rm -rf node_modules ...")
Labels were generated by AI models; expect weaker results on inputs written in a different format.