bge-small-sec-filings

BAAI/bge-small-en-v1.5 fine-tuned to retrieve passages from SEC 10-K filings (Business, Risk Factors, MD&A, Market Risk) for analyst-style questions. Built as the retrieval model for dealscope-agent.

Results

Held-out evaluation on 355 questions from companies never seen in training. Each query is ranked against every passage from those companies.

Metric	Base model	Fine-tuned	Change
ndcg@10	0.854	0.898	+0.043
mrr@10	0.816	0.868	+0.052
recall@1	0.730	0.797	+0.068
recall@10	0.972	0.989	+0.017

The correct passage is ranked first for 79.7% of questions instead of 73.0%, fixing about a quarter of the base model's top-1 misses. Test companies: Chevron, Johnson & Johnson, Meta, Microsoft, Oracle, Walmart.

Training

Data: jc4845/sec-10k-retrieval-qa, 1067 question-passage pairs. Questions are LLM-generated from the passage, then split by company to avoid leakage.
Loss: MultipleNegativesRankingLoss (in-batch negatives), no-duplicates batch sampler.
3 epochs, batch size 64, learning rate 2e-5, 10% warmup, fp16 on one NVIDIA T4 (about 30 seconds).
Queries use the bge instruction prefix: Represent this sentence for searching relevant passages:

Usage

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("jc4845/bge-small-sec-filings")
q = model.encode("Represent this sentence for searching relevant passages: What supply chain risks does Apple disclose?")

Limitations

Questions are synthetic (LLM-generated), so they are cleaner than real analyst queries.
Only US 10-K prose; tables, 10-Qs and non-US filings are not covered.
Small evaluation set; treat differences under about 0.02 as noise.

Loading…