LiLT-SER (Sinhala form understanding: semantic entity recognition)

This model labels each word of a Sinhala form as a QUESTION, ANSWER or HEADER, or as outside these entities. It uses IOB tags. It was released with the SinFUND dataset (SriDoc/datasets).

Model

• LiLT with an XLM-R base text encoder (nielsr/lilt-xlm-roberta-base), plus a token-classification head.
• Input: the words of a form and their bounding boxes, scaled to 0-1000. The model does not look at image pixels, so the words must come from an annotation or an OCR system.

Training

Sequential cross-lingual fine-tuning in two stages:

1. On the French part of XFUND (kavg/LiLT-SER-FR).
2. On the SinFUND training split (80 forms) for 10,000 steps, with a batch size of 8 and a learning rate of 5e-5.

Results

Results on the SinFUND test split (20 forms, 812 entities). The input was ground-truth words and boxes, and the metric is entity-level precision, recall and F1, micro-averaged over QUESTION, ANSWER and HEADER:

Important: SinFUND has no separate validation split. The "validation" scores in the training log below were measured on the SinFUND test split, and the released checkpoint was chosen by its test F1. The selected score is therefore optimistic. The final-step score involves no selection.

Limitations

• The scores assume perfect text and boxes. With OCR output, especially from handwriting, accuracy will be lower.
• The SinFUND test split shares templates with the training split (11 of 20 test forms). Results on completely new form layouts may be lower.

Usage

import torch
from transformers import AutoTokenizer, LiltForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("SriDoc/LiLT-SER")
model = LiltForTokenClassification.from_pretrained("SriDoc/LiLT-SER")

words = [...]   # words of the form
boxes = [...]   # one [x0, y0, x1, y1] box per word, scaled to 0-1000

enc = tokenizer(words, is_split_into_words=True, truncation=True, max_length=512, return_tensors="pt")
bbox = [[0, 0, 0, 0] if i is None else boxes[i] for i in enc.word_ids()]
with torch.no_grad():
    logits = model(input_ids=enc["input_ids"], attention_mask=enc["attention_mask"], bbox=torch.tensor([bbox])).logits
pred = logits.argmax(-1)[0].tolist()
# label of each word = prediction for its first sub-token
seen, labels = set(), []
for t, w in enumerate(enc.word_ids()):
    if w is not None and w not in seen:
        seen.add(w)
        labels.append((words[w], model.config.id2label[pred[t]]))

Forms with more than about 500 sub-word tokens must be split into chunks of 512 tokens.

Citation

@article{gunathilaka2025sinfund,
  title   = {SinFUND and SinOCR: Benchmarks for Sinhala Handwritten OCR and Template-Free Form Understanding},
  author  = {Gunathilaka, Kavishka and Hewagama, Danusha and Pushpakumara, Supul and Ambegoda, Thanuja D.},
  journal = {Research Square preprint},
  year    = {2025},
  doi     = {10.21203/rs.3.rs-6976719/v1}
}

Training log

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

The "Validation" columns were measured on the SinFUND test split (see above).

Framework versions

• Transformers 4.38.2
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.1