IT vs Non-IT TinyBERT Classifier, R98

Binary vacancy classifier for the first-stage gate: IT role vs Non-IT role.

This version is tuned for target recall IT >= 0.98, with a lower learning rate than the R99 baseline.

Model

• Base model: cointegrated/rubert-tiny2
• Architecture: BertForSequenceClassification
• Labels:
  • 0: NonIT
  • 1: IT
• Input text: title + " . " + description
• Description truncation: 2000 characters
• Max sequence length: 384

Training Data

Dataset: data/labeled/it_nonit_train_28_05_v2.csv

Rows:

Binary balance:

Training Setup

Command:

python3 classifier_agent/train_it_bert.py \
  --input data/labeled/it_nonit_train_28_05_v2.csv \
  --output-dir classifier_agent/it_vs_nonit_tiny_r98_lr2e5 \
  --device mps \
  --epochs 6 \
  --patience 2 \
  --batch-size 32 \
  --eval-batch-size 64 \
  --lr 2e-5 \
  --max-len 384 \
  --desc-limit 2000 \
  --target-recall 0.98 \
  --pos-weight-mult 1.0

Best checkpoint:

epoch 3

Validation Metrics

Threshold selected for target recall IT >= 0.98.

Confusion matrix at threshold 0.2934:

rows=true, cols=pred [NonIT, IT]

[[2452,  145],
 [  29, 1447]]

Error Analysis

False negatives: 29.

Main FN categories:

False positives: 145, all labeled Не IT.

Compared with the R99 baseline, this version trades recall for precision:

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL_DIR = "classifier_agent/it_vs_nonit_tiny_r98_lr2e5"
THRESHOLD = 0.2934192717075348

tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_DIR).eval()

def is_it(title: str, description: str = "") -> tuple[bool, float]:
    text = f"{title.strip()} . {' '.join(description.split())[:2000]}"
    enc = tokenizer(text, truncation=True, max_length=384, return_tensors="pt")
    with torch.no_grad():
        logits = model(**enc).logits
    proba_it = torch.softmax(logits, dim=-1)[0, 1].item()
    return proba_it >= THRESHOLD, proba_it

Recommendation

Use this version when reducing false positives is more important than catching the last 1% of weak or ambiguous IT roles. Keep the R99 model as a broader safety gate.