BERT-base feature-based adaptation on AG News (Topic Classification)

Model description

This model is a feature-based adaptation of bert-base-uncased for topic classification on the AG News dataset (4 classes: World, Sports, Business, Sci/Tech). The encoder is kept entirely frozen (0 BERT parameters trained); the [CLS] embedding from the last hidden state is used as a fixed feature vector, and a logistic regression classifier is trained on top of it.

This repo contains only the classifier (classifier.joblib). To run inference you need to:

1. Load bert-base-uncased frozen (AutoModel.from_pretrained, .eval(), no gradients).
2. Tokenize text the same way as in training (max_length=128).
3. Extract the [CLS] embedding from the last hidden state.
4. Feed that vector to the classifier in this repo (joblib.load("classifier.joblib")).

Training data

• Dataset: fancyzhx/ag_news
• Subsample used: 10,000 training examples, 2,000 test examples
• Max sequence length: 128 tokens

Training procedure

• Base model: bert-base-uncased (frozen, 0 trainable parameters)
• Adaptation method: Feature-based (CLS pooling + logistic regression)
• Feature extraction time: 91.5s
• Classifier training time: 46.1s

Evaluation results

Compared against a partial fine-tuning model (top 2 layers + head unfrozen), which achieved accuracy 0.9025 / F1 macro 0.9035. The fine-tuned model was chosen as the delivered model for this task based on this comparison; this feature-based model is kept as the rejected baseline with its own measurements.

Intended use

Topic classification of short English news headlines/snippets into one of four categories: World, Sports, Business, Sci/Tech. Intended as a reference baseline against fine-tuning, not as the primary delivered model for this task.

Limitations

• Trained on a 10k-example subsample of AG News, not the full training set.
• English only; performance on other languages or domains is not evaluated.
• Since the encoder is frozen, performance is capped by how well the pretrained embedding space already separates the classes; it underperforms fine-tuning on this task by about 3 points of accuracy.

References

• Devlin et al., 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Dataset: fancyzhx/ag_news