FinSight Insight 1.5B (LoRA adapter)

A LoRA fine-tune of Qwen2.5-1.5B-Instruct that rewrites structured personal-finance facts (spending insights, what-if savings simulations, savings-goal milestones) into short, warm, plain-text messages, using only the numbers it is given. It was built for the FinSight Android app, where a deterministic engine computes the numbers and the model only phrases them.

This repository holds the LoRA adapter for MLX. Ready-to-run quantized GGUF files are in itsonlyfranz/finsight-insight-qwen2.5-1.5b-GGUF. Training and evaluation data: itsonlyfranz/finsight-insight-synthetic.

Source code, training and evaluation scripts, and how to reproduce: github.com/itsonlyfranz/finsight-insight-llm.

How to run (mlx-lm)

pip install mlx-lm huggingface_hub
hf download itsonlyfranz/finsight-insight-qwen2.5-1.5b --local-dir finsight-adapter

from mlx_lm import load, generate

model, tokenizer = load("Qwen/Qwen2.5-1.5B-Instruct", adapter_path="finsight-adapter")

system = (
    "You are FinSight, a concise personal-finance coach for early-career earners. "
    "Use only the numbers provided; never invent amounts, rates, products or guarantees. "
    "All amounts are in Philippine pesos (₱); write them with the ₱ sign, never with a dollar sign. "
    "Plain text only, no markdown, no emojis."
)
user = """Monthly take-home income: ₱62,500.00
Insight type: risk nudge
Headline: Recurring Subscription Drift
Findings: Monthly recurring subscriptions currently sit at ₱3,737.29. Overlapping media or AI coding copilot tools add up fast.
Suggested action: Audit active tools and downgrade duplicate streaming or SaaS subscriptions to free up ~₱500-₱1,100/mo.
Amount at stake: ₱3,737.29

Rewrite the findings and the suggested action in a warm, specific voice for the user. Reply in exactly two lines:
INSIGHT: <at most two sentences>
ACTION: <one concrete sentence>"""

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": system}, {"role": "user", "content": user}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=160))

Or from the command line: mlx_lm.generate --model Qwen/Qwen2.5-1.5B-Instruct --adapter-path finsight-adapter --system-prompt "<system>" --prompt "<facts + instruction>" --temp 0.

Intended use and prompt format

Use a system message like the one above and a user message made of the facts block followed by one instruction. The three tasks use these instructions:

• insight: Rewrite the findings and the suggested action in a warm, specific voice for the user. Reply in exactly two lines:\nINSIGHT: <at most two sentences>\nACTION: <one concrete sentence>
• simulation: Explain in two or three sentences what these cuts mean for the user, in plain language.
• goal: Write one encouraging sentence (at most 30 words) about the next milestone for this goal.

Insight output is INSIGHT: ... / ACTION: .... Always validate the output with a number guard (every money figure must appear in the facts) and fall back to deterministic text on failure.

Out of scope: financial advice, tax or investment guidance, any input that is not in the facts format, other currencies or languages.

Training data

1,618 training / 114 validation examples of fully synthetic facts with templated reference texts (seed 20260512); see the dataset card. No real finances.

Training

Base Qwen2.5-1.5B-Instruct bf16; LoRA rank 16, scale 20, dropout 0.05, all attention and MLP projections, 28 layers (18.465M trainable parameters, 1.196%); AdamW, lr 2e-5 with 30-step warmup and cosine decay to 2e-6; batch 4, 600 iterations (~1.5 epochs of 1,618 examples), max sequence length 768; loss on the reply only; mlx-lm on an Apple M2 Pro (32 GB). Wall-clock 1,790 s including 7 validation passes; peak memory footprint 14.55 GB (unified memory). Validation loss 1.939 -> 0.077 (iter 100) -> 0.048 (200) -> 0.037 (300) -> 0.035 (400/500) -> 0.034 (600).

Evaluation

Held-out test set, n=246 (117 insight / 78 simulation / 51 goal); greedy decoding; M2 Pro 32 GB.

Number fidelity = pass rate of a port of the app's money-figure guard. Format valid = raw output follows the app's output contract. ROUGE-L is measured against the templated reference. Fact coverage = share of reference money figures the output states. tok/s and latency: MLX rows via mlx-lm, GGUF rows via llama-server (llama.cpp Metal) over HTTP, so the groups are different runtimes. Fine-tuned MLX is slower than base because the LoRA adapter is unfused at runtime.

Zero-shot, the base model follows the output contract on 58.9% of prompts; the fine-tuned model on 100%, with ROUGE-L 0.454 vs 0.252 (base) and 0.373 (2-shot prompt).

Limitations and honest notes

• References are templated (not human-written), so ROUGE-L measures closeness to a house style. No LLM-as-judge was run (no independent judge model offline; judging with the model under test is circular).
• Base model number fidelity was already high (97.6%; 6 of 246 failures: 5 simulation, 1 insight); fine-tuning's main wins are format reliability, quality vs reference, and not needing a long few-shot prompt.
• Single run/seed, no confidence intervals; tests come from the same generator as training; no Android on-phone measurement was made.
• Trained and tested only on synthetic data from one generator; real-world robustness is unmeasured. Number fidelity of 100% on 246 prompts is not a guarantee; keep the guard.
• A first run at lr 1e-4 diverged (loss 0.94 -> 4.5-5.0); it was restarted at 2e-5. A first full pass exposed a flaw in the author's own dataset (two-sentence goal references vs the app's one-sentence contract); it was fixed, regenerated, and everything re-run. Reported numbers are from the corrected data only.
• Output wording is repetitive by design (few phrasings per case).
• Not financial advice. May state generic recommendations that came from the facts (for example savings-account yield text) without verifying them.

License

Apache-2.0, inherited from the base model Qwen2.5-1.5B-Instruct.