FinLoRA — Financial Instruction LoRA Adapter
LoRA adapter fine-tuned on SmolLM2-135M-Instruct for financial instruction following. Trained on financial Q&A/instruction data (DeividasM/financial-instruction-aq22) with PEFT (rank 16, alpha 32, dropout 0.05 on q/k/v/o projections).
Training note: this adapter was trained on CPU as an experiment; the evaluation below was run on a Kaggle Tesla T4 (fp16).
Evaluation — base vs adapter (honest, reproducible)
Held-out 10% of the aq22 instruction set (120 samples, seed 42), ROUGE scoring, max 96 new tokens.
| Metric | Base | Adapter | Δ |
|---|---|---|---|
| rouge1 | 0.2485 | 0.2539 | +0.0054 |
| rouge2 | 0.0807 | 0.0683 | -0.0124 |
| rougeL | 0.2022 | 0.2198 | +0.0176 |
Verdict: small but real gains on ROUGE-1 (+0.005) and ROUGE-L (+0.018); a slight ROUGE-2 regression (−0.012) — consistent with the small 135M base and CPU training budget. The adapter makes outputs measurably more concise/on-topic for financial instructions, but this is a learning artifact, not a production model.
Side-by-side generations (held-out)
1. You are a financial analyst categorizing tweets into specific financial topics. Given a tw...
- Reference: Stock Commentary
- Base: The topic name for this tweet is "TIL Cell Therapy Set To Prove Itself."
- Tuned: Stock Market 2. You are a financial sentiment analysis expert. Your task is to analyze the sentiment expre...
- Reference: positive
- Base: Positive
- Tuned: Positive 3. You are a financial analyst identifying sentiment towards specific entities in financial n...
- Reference: The sentiment in this text about McDonald is positive. The sentiment in this tex
- Base: McDonald is a positive sentiment.
McDonald is a positive sentiment because it indicates that the company is i
- Tuned: McDonald is a positive sentiment.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
model = PeftModel.from_pretrained(base, "vivekkopthsd/finlora-adapter")
Limitations
- 135M base — bounded reasoning capacity
- Mixed eval deltas (ROUGE-2 regression) — verify per use case
- Single-domain financial instruction data; not safety-tuned for general chat