Screenshot Intent Classifier

This repository contains a Longformer-based classifier fine-tuned to decide whether a conversational agent should trigger a screenshot tool for the latest user message.

Base Model

This model is fine-tuned from allenai/longformer-base-4096, and inherits the base encoder's maximum context length and tokenizer. "longformer-base-4096" is a RoBERTa-style encoder with support for sequences up to 4,096 tokens using a combination of sliding-window local attention and user-configured global attention. We follow the common pattern for classification tasks with Longformer by assigning global attention to the first token in each sequence.

Classifier

• 0 / no_screenshot: do not call the screenshot tool.
• 1 / take_screenshot: call the screenshot tool.

The input is a text block representing the recent conversation history, formatted as one utterance per line, prefixed with a speaker tag, e.g.:

USER: I'm wondering if blue goes well with yellow.
USER: What's your take on this?

At inference time, the host application typically feeds the last few conversation turns (most importantly the latest user message) in this format and thresholds the classifier's take_screenshot probability to decide whether to trigger the tool.

Training Data

The classifier was trained on a curated, hand-labelled private dataset. It contains hundreds of single-turn and multi-turn examples specifying whether each user message should or should not trigger a screenshot, including:

• Clear positive triggers ("look at this", "check this out", "rate this pic").
• Clear negatives (off-topic chit-chat, abstract statements, idioms like "I'll look into it").
• Edge cases involving deictic pronouns, quantities ("take 2 screenshots"), negation ("don't look"), multi-turn context, and more.

No external user logs or third-party datasets were used; the training data is purely synthetic / curated for this intent task.

Training Setup

Approximate defaults:

• Epochs: 3
• Batch size: 16 (per device)
• Learning rate: 2e-5
• Weight decay: 0.01
• Max sequence length: 1536 tokens (truncation for old entries applied beyond this)

The script builds examples by concatenating conversation history up to and including the current user message, one utterance per line prefixed with "USER:". Multi-turn conversations therefore become multiple training examples with growing context.

Usage

Basic usage with the Transformers library:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

MODEL_ID = "yapwithai/yap-longformer-screenshot-intent"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
model.eval()

text = "USER: look at this amazing sunset"
inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    padding="max_length",
    max_length=1536,
)

attention_mask = inputs["attention_mask"]
global_attention_mask = torch.zeros_like(attention_mask)
global_attention_mask[:, 0] = 1

with torch.no_grad():
    outputs = model(**inputs, global_attention_mask=global_attention_mask)
    probs = outputs.logits.softmax(dim=-1)[0]

p_no, p_yes = probs.tolist()
print("P(no_screenshot)=", p_no)
print("P(take_screenshot)=", p_yes)

In production, you would:

• Construct a conversation history string similar to the training format (recent user turns, optionally assistant turns, each on its own line with a speaker prefix).
• Run the classifier once per latest user message.
• Threshold p_yes to decide whether to trigger the screenshot tool.

Longformer Citation

If you use Longformer in your work, please cite:

@article{Beltagy2020Longformer,
  title={Longformer: The Long-Document Transformer},
  author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
  journal={arXiv:2004.05150},
  year={2020},
}

Longformer is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).