Loading…
Loading…
Fine-tuned distilbert-base-uncased that reads a patent abstract and predicts its CPC section — the top, broadest level of the Cooperative Patent Classification hierarchy (a single letter, A–H plus Y; 9 classes).
"Section", not the whole CPC code. CPC is hierarchical — section (
G, 9 classes) → class (G06, ~130) → subclass (G06F, ~660) → group/subgroup. This model predicts only the section (the first letter), which is the coarsest and easiest level.
| id | CPC | Section |
|---|---|---|
| 0 | A | Human Necessities |
| 1 | B | Performing Operations; Transporting |
| 2 | C | Chemistry; Metallurgy |
| 3 | D | Textiles; Paper |
| 4 | E | Fixed Constructions |
| 5 | F | Mechanical Engineering; Lighting; Heating; Weapons; Blasting |
| 6 | G | Physics |
| 7 | H | Electricity |
| 8 | Y | General tagging of new or cross-sectional technology |
from transformers import pipeline
clf = pipeline("text-classification", model="crismolav/distilbert-patent-cpc-classifier", top_k=3)
clf("a steam turbine exhaust hood includes an exhaust hood section and a butterfly "
"plate arranged in the exhaust hood section with a complex curvilinear profile.")
# -> [{'label': 'Mechanical Engineering; ...', 'score': ...}, ...]
Macro-F1 is the headline metric because the classes are heavily imbalanced (~26× between the largest and smallest section), so plain accuracy is misleading.
| Model | Macro-F1 | Accuracy |
|---|---|---|
| TF-IDF + Logistic Regression (baseline) | 0.558 | 0.594 |
| DistilBERT (this model) | 0.613 | 0.667 |
In context — on the same dataset, the LSG Attention paper (Table 2) reports RoBERTa-512 at 0.618 macro-F1 / 0.666 micro-F1, and the best long-document model (LSG) at ~0.644 macro-F1. This model matches the RoBERTa benchmark using a smaller, distilled architecture on abstracts truncated to 256 tokens — i.e. it sits ~3 points below the dataset's practical ceiling.
Strongest: Human Necessities (0.76), Electricity (0.75), Physics (0.72).
Weakest by far: CPC Y (0.11) — "general / cross-sectional technology". Y is applied on top of a primary technical area (e.g. a climate-friendly chemistry patent is tagged both C and Y), so as a single-label target it overlaps every other section and is intrinsically hard.
CPC is inherently multi-label (a patent carries ~1–3 codes on average and often spans several sections); this dataset keeps only the primary one. So many top-1 "errors" are the model naming a genuinely valid co-section rather than misreading the patent. On the test split:
| top-1 | top-2 | top-3 | |
|---|---|---|---|
| Overall | 0.667 | 0.843 | 0.927 |
Class Y |
0.072 | 0.732 | 0.978 |
Y is ranked #1 only ~7% of the time but sits in the top-2 ~73% / top-3 ~98% — the model recognises Y patents, it just ranks their concrete technical section (C, F, H…) above the cross-cutting Y tag. Much of the apparent error is the single-label reduction of a multi-label task, not the model being wrong.
ccdv/patent-classification, abstract config (derived from BIGPATENT). Official splits: 25k / 5k / 5k.
The training split was cleaned for integrity before fine-tuning:
Net training set: 24,660 abstracts (the 340 removed rows are ~1.4% of train).
distilbert-base-uncasedAn inverse-frequency ("balanced") class-weighted cross-entropy was also tried to combat the imbalance. On test it was roughly a wash for macro-F1 (0.614 vs 0.613) at a small accuracy cost, but it more than doubled the worst class (Y F1 0.11 → 0.21, recall 0.07 → 0.18). The takeaway: inverse-frequency weighting redistributes performance toward rare classes rather than strictly improving the mean, and Y's difficulty is semantic overlap, not frequency. This published checkpoint is the unweighted model (higher accuracy, equal macro-F1).
Y section is unreliable (see above) — treat low-confidence Y predictions with caution.ccdv/patent-classification.distilbert/distilbert-base-uncased