Qwen2-VL-2B receipt extraction (LoRA adapter)
A LoRA adapter for Qwen2-VL-2B-Instruct that reads a receipt image and returns JSON with the line items and the total. Trained with QLoRA on a single Kaggle T4 GPU.
Code and evaluation: github.com/Chaithra3-6/qwen2vl-receipt-extraction
Results (60 held-out receipts)
| Metric | Score |
|---|---|
| Valid JSON output | 100% |
| Total exact-match | 90% |
| Mean character similarity of the total field (all 60) | 0.957 |
- The 60 receipts are the first 60 of CORD-v2's validation split. They were not used for training; training ran a fixed 2 epochs with no checkpoint selection.
- Only the total field is scored. Item names and prices are not scored.
- Similarity is 1 minus normalized Levenshtein distance on the total, averaged over all 60 receipts including exact matches. The 6 misses average about 0.57.
Prompt and output
Prompt used in training and evaluation:
Extract the receipt into JSON with keys: items (a list of name and price) and total. Output only JSON.
Output format:
{"items": [{"name": "EGG TART", "price": "13,000"}], "total": "13,000"}
Usage
Load with Unsloth, the same way the model was trained and evaluated:
from unsloth import FastVisionModel
from PIL import Image
model, tokenizer = FastVisionModel.from_pretrained(
"chai1me/qwen2vl-receipt-lora", load_in_4bit=True)
FastVisionModel.for_inference(model)
INSTR = ("Extract the receipt into JSON with keys: items (a list of name and price) "
"and total. Output only JSON.")
image = Image.open("receipt.jpg").convert("RGB")
image.thumbnail((1024, 1024))
messages = [{"role": "user", "content": [
{"type": "image", "image": image}, {"type": "text", "text": INSTR}]}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.batch_decode(
out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
Training details
| Setting | Value |
|---|---|
| Base model | unsloth/Qwen2-VL-2B-Instruct-bnb-4bit (4-bit) |
| LoRA targets | language model attention and MLP layers; vision layers frozen |
| Rank / alpha / dropout | 16 / 16 / 0 (no bias) |
| Trainable parameters | 18,464,768 of 2,227,450,368 (0.83%) |
| Data | CORD-v2, first 400 usable training receipts, images at most 1024 px |
| Epochs / steps | 2 / 100 |
| Batch size | 2, with 4 gradient-accumulation steps (effective 8) |
| Learning rate | 2e-4, linear decay, 5 warm-up steps |
| Optimizer | 8-bit AdamW, weight decay 0.01 |
| Seed | 3407 |
Limitations
- Trained on CORD (cafe and restaurant receipts, mostly Indonesian); other document types need retraining.
- Small evaluation set (60), one training run, and only the total field is scored.
- Known item-level errors: dropped zero-price lines and changed currency spacing
(
Rp29,090vsRp 29,090). - Field extraction only; not a forgery or tamper detector.