BERT sensitive-product classifier

Fine-tuned bert-base-uncased that classifies an Amazon product as sensitive (buying it implies a medical condition or health concern) or nonsensitive. It is the on-device obfuscator of the paper Hybrid Obfuscation and On-Device Deobfuscation for Privacy-Preserving LLM-Based Product Recommendation: products it flags stay on the user's device and are not sent to a server-side LLM recommender.

Code: https://github.com/tinakhezresmaeilzadeh/privacy-preserving-llm-rec · Paper: https://arxiv.org/abs/2505.00951

Usage

The input is the product's title, main category and features in the format below (bert_text in privrec/obfuscators.py of the code repository). The paper truncates inputs to 128 tokens and flags a product if P(sensitive) > 0.5.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = "tina-khezresmaeilzadeh/bert-sensitive-product-classifier"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

def text(title, main_category, features):
    s = f"{title}**\n- Main Category: {main_category or 'N/A'}\n"
    s += ("- Features:\n" + "".join(f"  - {f}\n" for f in features)) if features else "- Features: N/A\n"
    return s + "\n"

x = [text("Blood Glucose Monitor Kit", "Health & Personal Care", ["For diabetes testing"]),
     text("Stainless Steel Water Bottle 32oz", "Sports & Outdoors", ["Insulated"])]
with torch.no_grad():
    p = torch.softmax(model(**tok(x, return_tensors="pt", padding=True, truncation=True, max_length=128)).logits, -1)[:, 1]
print(p.tolist())          # about [0.95, 0.11]; sensitive if > 0.5

Training

• Data: 10,012 products sampled from the Amazon Reviews 2023 categories Health & Household, Health & Personal Care and Beauty & Personal Care, labelled by GPT-4o with the few-shot prompt in the paper (product metadata only; 1,490 sensitive). The labels are in data/obfuscator_training_labels.jsonl of the code repository.
• Procedure: class-weighted focal loss (gamma = 2), 5 epochs, learning rate 2e-5, weight decay 0.01, batch size 16, inputs truncated to 256 tokens; 70/20/10 split stratified by label; checkpoint selected by validation F1.

Evaluation

• Validation split (used for checkpoint selection, so optimistic): F1 0.868, precision 0.815, recall 0.928.
• Paper's evaluation (25,899 purchase-history entries of 1,295 users across ten categories, compared with independent Qwen3.5-9B labels under the same written criteria): the classifier flags 17% of entries, with precision 0.42 and recall 0.75 (F1 0.54). It flags 91% of the strictly sensitive entries (products that treat, monitor or manage a specific condition). Deleting the three health and beauty categories instead reaches F1 0.33.

Limitations

The notion of "sensitive" is the one in the labelling prompt, which includes everyday symptom-relief products; in a blinded human check of 200 products, a human annotator applied a narrower notion. The model was trained on three health and beauty categories only, and the products it misses are mostly mildly sensitive. It classifies products one at a time, so it does not prevent a server from inferring a condition from the products that are sent.

Citation

@article{khezresmaeilzadeh2025hybrid,
  title   = {Hybrid Obfuscation and On-Device Deobfuscation for Privacy-Preserving {LLM}-Based Product Recommendation},
  author  = {Khezresmaeilzadeh, Tina and Zhang, Jiang and Andreadis, Dimitrios and Psounis, Konstantinos},
  journal = {arXiv preprint arXiv:2505.00951},
  year    = {2025}
}