Advocate Remuneration Assistant: SFT adapter (epoch 1)

A LoRA adapter for Qwen/Qwen2.5-7B-Instruct, fine-tuned with QLoRA (supervised fine-tuning) for a retrieval-augmented assistant that answers questions about Kenya's Advocates (Remuneration) Order and Kenyan case law on taxation of costs.

This is the supervised fine-tuning stage. The follow-up on-policy DPO adapter is mikekeror/qwen2.5-7b-advocate-dpo-lora.

Not legal advice. This is a research prototype. Figures reflect the Order as revised to 2022 and may not reflect later amendments or a taxing officer's discretion.

Read this first: what this adapter does and does not do

In a pre-registered evaluation on a frozen 236-question held-out set, this adapter did not show a detectable accuracy gain over the base model with the same retrieval pipeline (SFT vs base: -2.5 points, 95% interval [-5.9, +0.9]). The accuracy gains in this project came from the retrieval and calculation pipeline around the model, not from fine-tuning. On an earlier 62-question set, fine-tuning did remove fabricated citations (11 to 0).

Full write-up: https://mikekeror-rgb.github.io/advocate_renumeration_assistant_v1/report.pdf

How it is meant to be used

The adapter is trained for use inside the full pipeline, not on its own: hybrid retrieval (dense + BM25 + reciprocal rank fusion), case-scoped retrieval when a question names a ruling, and a deterministic fee calculator whose result the model is told to state. Prompts contain numbered source chunks, and the model is trained to cite them and to say "Not covered in the provided documents" when the sources do not answer.

Used without retrieved context, it has no reliable legal knowledge of its own. Code for the pipeline: https://github.com/mikekeror-rgb/advocate_renumeration_assistant_v1

Usage

With transformers and peft:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto", torch_dtype="auto")
model = PeftModel.from_pretrained(model, "mikekeror/qwen2.5-7b-advocate-sft-lora-ep1")

With vLLM (multi-LoRA, base and adapters served side by side):

vllm serve Qwen/Qwen2.5-7B-Instruct --enable-lora \
  --lora-modules advocate-ep1=mikekeror/qwen2.5-7b-advocate-sft-lora-ep1

Training

• Data: 550 synthetic examples: 300 grounded/ruling, 100 grounded/statute, 60 calculator, 60 refusal, 30 off-scope. Generated with gpt-oss-20b run locally. The teacher never writes citations; a script inserts them so the format is verifiable. The train/validation split is grouped by case, and every evaluation case, passage, question and amount is excluded from training.
• Method: QLoRA (4-bit NF4), LoRA r=16 on all linear layers, 40.4M trainable parameters (0.53%), completion-only loss.
• Run: 2 epochs on one RTX 4090 in 42.7 minutes. Validation loss was lowest after epoch 1 (0.198 vs 0.206), so epoch 1 is the adapter released here.

Evaluation (frozen 236-question held-out set, pipeline fixed)

Accuracy, percent, judged against reference answers (numeric answers checked deterministically):

Judge faithfulness (1 to 5): base 4.59, this adapter 4.59. Differences between base and SFT are within the noise of this test. A rerun of the base model changed 2 of 236 answers. The judge (gpt-5.4-mini) was hand-checked on 30 verdicts, with 30 of 30 agreeing.

Limitations

• One domain, one base model, one language pair of corpus and questions. Do not generalise from this to other settings.
• Training data is synthetic, written by a 20B teacher and filtered by rules. It was not reviewed by a lawyer.
• The statute question group in the evaluation has only 23 questions.
• The adapter should not be used to give legal advice.

Citation

Keror, M. (2026). Advocate Remuneration Assistant: pipeline engineering vs fine-tuning (software and evaluation). https://doi.org/10.5281/zenodo.23603007