TinyLlama 1.1B Financial Form Normalizer LoRA

A task-conditioned English form-normalization adapter by sraivante, fine-tuned on 3,657 examples spanning 38 tasks. It converts short user replies into application values such as dates, amounts, ZIP codes, booleans, intent labels and small JSON objects, using the matching task-specific system prompt.

This is a 4.52 MB LoRA adapter, with 2,252,800 trained parameters, for TinyLlama/TinyLlama-1.1B-Chat-v1.0. It requires the approximately 1.1B-parameter base model; those full weights are downloaded separately. The published adapter is unchanged from the final local checkpoint, step 1,374.

The source project is GOOGLE_COLAB/FINE_TUNE/tinyllama_fine_tuned, with its training notebook in LLM_FINE_TUNE/finetune/colab_code/finetuning_colab.ipynb and data in fp_finetune_jsonl/chat/all_combined.jsonl.

Intended behavior

Use a task's bundled system prompt from task_prompts.json, followed by the user's short reply. Examples of task families:

• Dates, ZIP codes, retirement ages, savings amounts and numeric field values.
• Yes/no, tobacco-use, tax-status and user-supplied health-profile labels.
• Healthcare plan names, selection states and profile-editing intents.
• Navigation, illustration selection and a small stage/state JSON format.

Return formats differ by task: a number, a date such as 10-1986, a boolean, an exact label, RETRY, NONE, NULL or a small JSON object. Preserve the task prompt; the user message alone does not identify the intended mapping.

The financial and healthcare constants in these prompts are fixed application examples, not current tax/Medicare rules, forecasts, eligibility decisions or personalized advice. Validate outputs and current business rules separately before using them in an application. The model has not been evaluated as a general financial or medical assistant.

Load the adapter

The release verification used PyTorch 2.12.0, Transformers 4.56.2, PEFT 0.17.1 and Tokenizers 0.22.1, with unquantized BF16 base weights on CPU. The training environment used newer library versions, listed below. PEFT 0.17.1 warns about extra inactive metadata in the original adapter configuration; exact loading of every adapter tensor was verified.

pip install "torch>=2.6" "transformers==4.56.2" "peft==0.17.1" accelerate safetensors

import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

adapter_id = "sraivante/TinyLlama-1.1B-Financial-Form-Normalizer-LoRA"
base_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
base_revision = "fe8a4ea1ffedaf415f4da2f062534de366a451e6"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id, revision=base_revision,
    torch_dtype=torch.bfloat16, low_cpu_mem_usage=True,
)
model = PeftModel.from_pretrained(
    base, adapter_id, is_trainable=False, autocast_adapter_dtype=False,
).eval()

with open(hf_hub_download(adapter_id, "task_prompts.json"), encoding="utf-8") as f:
    task_prompts = json.load(f)

messages = [
    {"role": "system", "content": task_prompts["birthdate"]["system"]},
    {"role": "user", "content": "My birth month was October in the year 1986."},
]
# Match the original inference notebook: template first, tokenizer second.
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt")
with torch.inference_mode():
    outputs = model.generate(
        **inputs, max_new_tokens=32, do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(
    outputs[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
).strip())

See example_normalize.py for a command-line version with a task selector and context-length check. The base model and saved tokenizer support a 2,048-token context. Keep the task prompt, input and output within that limit. The saved adapter has no GGUF or merged-weight export in this release.

The base revision above is pinned for release verification. The original adapter configuration records the base name but no revision, so the exact base commit used during training is not independently recorded.

Training record

The notebook defines a separate MAX_SEQ_LENGTH=512 variable, but the saved SFT configuration says max_length=1024 and the final tokenizer says 2,048. Re-tokenizing the packaged training messages produced lengths of 64–302 tokens, below all three limits. The notebook calls model-level gradient checkpointing; the saved trainer argument flag itself is false.

Warmup ratio 0.03 appears in the notebook and serialized arguments. The newer serialized warmup_steps field also contains 0.03; no precise effective warmup schedule is reconstructed. The logged learning rate stays at 0.0002.

The saved setup cell reports a Tesla T4 and the later training cell reports BF16 enabled. No independent device profile was archived. The notebook records completion at 1,374/1,374, epoch 3/3, with elapsed time around 1 hour 43 minutes.

The original card reports: PEFT 0.19.1, TRL 1.4.0, Transformers 5.9.0, PyTorch 2.10.0+cu128, Datasets 4.8.5 and Tokenizers 0.22.2. See training/training_config.json, training/trainer_state.json and the source notebook. The notebook's code is retained; outputs and Colab execution metadata were removed from the published copy. Optimizer and pickle training-state files are omitted.

Evaluation

The final logged training loss is 0.108085 at step 1,350. The final checkpoint is step 1,374. This is training loss, not validation loss or field-level accuracy; repeated system-prompt tokens contribute to training metrics.

A paired release diagnostic compared the unchanged base with the adapter on 12 newly authored inputs absent verbatim from the packaged training corpus. Both used the same task system prompts and deterministic generation capped at 32 new tokens.

The adapter returned RETRY for the jointly filing tax-status example, whose expected label was MARRIED_FILING_JOINTLY. The base generally returned explanations instead of the required bare values. Only surrounding whitespace was removed for scoring; this measures strict value-and-format matching on these twelve inputs.

This is a small convenience sample across 11 tasks, not an independently curated benchmark or evidence of reliability across all 38 tasks. Use evaluate_adapter.py to repeat the diagnostic. Full prompts, expected labels, raw outputs and conditions are in verification.json and the linked dataset. No data was used to tune the adapter during this release.

All 88 adapter tensors loaded exactly; finite values, nonzero LoRA matrices, tokenizer behavior and the base-weight checksum were checked. The checkpoint's hash matches the final saved checkpoint-1374 adapter.

Data provenance and known limitations

The Financial Form Normalization Instructions dataset contains the training JSONL, existing project diagnostic and new release diagnostic. An identical copy is included in training_data/.

The local combined chat corpus and the May 20 uploaded copy have identical parsed records and order; their whitespace/serialization bytes differ. The notebook's filename and recorded 3,657-example count match this corpus. No training-time dataset checksum was recorded, so historical byte identity remains unproven.

There are 12 conflicting system/user prompt pairs with different target labels. The original 114-example project test file overlaps training in every case, so it is labeled as a diagnostic, not a held-out test set. There is no genuine training-time validation or test split.

The data contains short, example-style field values and typo variants. Its exact authoring/generation process was not archived, and no external dataset attribution accompanies this corpus. No additional cleaning, label correction or deduplication was performed for publication.

Expect errors on ambiguous replies, unsupported task prompts and unfamiliar values. Fixed ZIP-code defaults, narrow gender labels and other application-specific assumptions may be unsuitable elsewhere. The model can emit wrong labels or extra text. Application schema checks, ambiguity handling and user confirmation of consequential values remain necessary.

License and attribution

Copyright (c) 2026 sraivante for original fine-tuning contributions, original dataset contributions and original release material, to the extent copyright subsists. These are released under Apache License 2.0.

TinyLlama's base model is separately provided by the TinyLlama project under Apache-2.0. Preserve upstream attribution; this release does not claim ownership of the base model, its tokenizer or its chat template. See NOTICE, upstream model card and the dataset's copyright statement.

Release prepared 2026-09-25.