Fin-ModernBERT

Fin-ModernBERT is a domain-adapted pretrained language model for the financial domain, obtained by continual pretraining of ModernBERT-base with a context length of 1024 tokens on large-scale finance-related corpora.

---

Model Description

• Base model: ModernBERT-base (context length = 1024)
• Domain: Finance, Stock Market, Cryptocurrency
• Objective: Improve representation and understanding of financial text for downstream NLP tasks (sentiment analysis, NER, classification, QA, retrieval, etc.)

---

Training Data

We collected and combined multiple publicly available finance-related datasets, including:

• danidanou/Bloomberg_Financial_News
• juanberasategui/Crypto_Tweets
• StephanAkkerman/crypto-stock-tweets
• SahandNZ/cryptonews-articles-with-price-momentum-labels
• edaschau/financial_news
• sabareesh88/FNSPID_nasdaq
• BAAI/IndustryCorpus_finance
• mjw/stock_market_tweets

After aggregation, we obtained ~50M financial records.
A deduplication process reduced this to ~20M records, available at:
👉 clapAI/FinData-dedup

---

Training Hyperparameters

The following hyperparameters were used during training:

• Learning rate: 2e-4
• Train batch size: 24
• Eval batch size: 24
• Seed: 0
• Gradient accumulation steps: 128
• Effective total train batch size: 3072
• Optimizer: AdamW_Torch_Fused with betas=(0.9, 0.999), epsilon=1e-08
• LR scheduler: Linear
• Epochs: 1

---

Evaluation Benchmarks

We benchmarked Fin-ModernBERT against two strong baselines:

• ProsusAI/finbert
• answerdotai/ModernBERT-base

Fine-tuning Setup

All models were fine-tuned under the same configuration:

• Optimizer: AdamW
• Learning rate: 5e-5
• Batch size: 16
• Epochs: 5
• Scheduler: Linear

Results

Further evaluations on additional datasets and tasks are ongoing to provide a more comprehensive view of its performance.

---

Use Cases

Fin-ModernBERT can be used for various financial NLP applications, such as:

• Financial Sentiment Analysis (e.g., market mood detection from news/tweets)
• Event-driven Stock Prediction
• Financial Named Entity Recognition (NER) (companies, tickers, financial instruments)
• Document Classification & Clustering
• Question Answering over financial reports and news

---

How to Use

from transformers import AutoTokenizer, AutoModel

model_name = "clapAI/Fin-ModernBERT"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

text = "Federal Reserve hints at possible interest rate cuts."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

Citation

If you use this model, please cite:

  title={Fin-ModernBERT: Continual Pretraining of ModernBERT for Financial Domain},
  author={ClapAI},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/clapAI/Fin-ModernBERT}}
}