MiniLM-L6-v2 LoRA — Tax & Finance Retrieval

A parameter-efficient retrieval encoder fine-tuned from sentence-transformers/all-MiniLM-L6-v2 for Indian tax and finance question-answering retrieval.

The model maps a taxpayer's question to the exact statute passage (or financial document) it is answered by. It is compact (22M params, ~90 MB), fast to embed, and cheap to deploy — designed as a retrieval unit for a RAG pipeline on modest hardware.

Key facts

Property	Value
Base model	`all-MiniLM-L6-v2` (22.7M params)
Fine-tuning	LoRA (r=16, α=32, dropout 0.05) on query/key/value/dense
Trainable params	675,840 — 2.89% of the model
Embedding dim	384
Max sequence length	256 tokens
Loss	MultipleNegativesRankingLoss
Compute	GPU (Kaggle), P100
License	Apache-2.0 (base model)

Headline results

Measured on a held-out pooled retrieval test of 839 queries scored against the full combined tax + finance corpus (100% unseen during training). Reported as mean ± std across 3 independent training seeds (42, 123, 456).

Metric	LoRA fine-tuned	Zero-shot baseline	Δ
nDCG@10	0.592 ± 0.001	0.447	+0.145
MRR@10	0.484 ± 0.002	0.361	+0.123
MAP@100	0.488 ± 0.002	0.373	+0.115
Recall@1	0.327 ± 0.002	0.250	+0.076
Recall@5	0.685 ± 0.001	0.496	+0.189
Recall@10	0.948 ± 0.002	0.735	+0.212
Accuracy@1	0.327 ± 0.002	0.250	+0.076

Fine-tuning delivers a +32% relative improvement in nDCG@10 over the untrained base, with high seed-to-seed stability (std ≤ 0.002).

Eval protocol (research-grade)

Stratified splits (by tax topic) — train / val / test = 6,708 / 838 / 839; no leakage.
Elbow-based epoch selection on the validation set only; the test set is scored exactly once.
3 seeds, aggregate mean ± std.
Zero-shot baseline on the identical held-out test.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("vivekkopthsd/minilm-l6-v2-taxfinance-lora")

query = model.encode(
    ["What is the deduction for rent paid on salary income?"],
    normalize_embeddings=True,
)
docs = model.encode(
    [
        "Section 134: Deduction in respect of rent paid.",
        "Section 123: Deduction for life insurance premium.",
    ],
    normalize_embeddings=True,
)

scores = query @ docs.T

Training

Data — pooled from two sources:

ligaments-dev/indian-income-tax-qa — 6,800 (question, answer) pairs from Indian Income-Tax statutes, used as (query, positive-document) pairs.
AIR-Bench/qa_finance_en (AIR-Bench_24.04) — 1,585 financial texts, used as self-supervised pairs.

Objective — MultipleNegativesRankingLoss: for each (query, positive) pair in a batch, the positive is treated as the correct answer while all other documents in the batch act as in-batch negatives. This is the standard contrastive objective for dense retrieval.

Hyperparameters

Parameter	Value
Optimizer	AdamW (weight decay 0.01)
Learning rate	2e-4
Batch size	32
Scheduler	cosine with 10% warmup
LoRA rank / α	16 / 32
LoRA dropout	0.05
Epochs	15 (elbow-selected ~4–5)
Max grad norm	1.0
Seeds	42, 123, 456

Limitations

English only. The base model has limited multilingual capacity; retrieval quality degrades sharply for Hindi or code-mixed Hinglish input. For multilingual retrieval, a multilingual-base fine-tune is required.
Domain-specific. Trained on Indian income-tax Q&A and generic finance; generalization to unrelated legal or technical domains is not guaranteed.
Short context. 256-token limit suits statute sections and FAQ-style documents, not long passages.
The finance subset uses self-supervised (text→text) pairs, so pooled metrics should be interpreted alongside the tax-only performance.

Citation

@misc{vivekkopthsd-minilm-l6-v2-taxfinance-lora,
  title  = {MiniLM-L6-v2 LoRA: Tax and Finance Retrieval},
  author = {Vivek},
  year   = {2026},
  url    = {https://huggingface.co/vivekkopthsd/minilm-l6-v2-taxfinance-lora}
}