DistilBERT fine-tuned on arXiv abstracts (6 subjects)

NLRAI CA-IV Project #32 · Mohammad Jamal Yusuf · SICSR, Pune.

Fine-tuned distilbert-base-uncased that classifies a research abstract into one of six subjects: Natural Language Processing, Computer Vision, Cryptography & Security, Statistical Machine Learning, Neuroscience, Astrophysics (Galaxies).

• Data: 6,000 arXiv abstracts (1,000 per subject, primary category as label), collected 10 Oct 2026; split 70/15/15, seed 42.
• Training: 3 epochs, AdamW, lr 3e-5, 10% warm-up, effective batch 16, max length 256 tokens.
• Test (900 abstracts): accuracy 0.918, macro-F1 0.918. A TF-IDF + Logistic Regression baseline scored 0.940 on the same split.
• Known limitation: 67.5% of abstracts exceed 256 tokens and are truncated.

Live demo: https://huggingface.co/spaces/jamal10110/academic-document-classifier