DocFields-F32-KO
🇰🇷 한국어 카드: MODEL_CARD.ko.md
Status: EXPERIMENTAL / RESEARCH PREVIEW. Quality verdict:
F32_TARGETS_MISSED— this checkpoint met 6 of its 7 pre-registered development targets and missed the phone-number target. It is published as a research artifact, not as a product. There is no accuracy guarantee for production documents.
DocFields-F32-KO labels the OCR boxes of a Korean financial form (claim forms, applications, confirmations) with a field role: address, phone number, account number, claim amount, issue date, other dates and so on. It takes text boxes and their coordinates; it does not read pixels and it does not include OCR. It is a full fine-tune of LiLT + XLM-RoBERTa with a 15-label token-classification head.
What F3.2 changed over its predecessor: one checkpoint now recovers generic dates and account numbers that the previous single model mostly dropped, without calling a second model or a rule engine at inference time.
0. How to use
Input is a list of OCR boxes for one page. bbox is [x0, y0, x1, y1] normalized to 0..1 (x to the right, y down).
pip install -r requirements.txt
python docfields_f32.py --selftest
python docfields_f32.py --model . --example examples/synthetic_form.json
from huggingface_hub import snapshot_download
import sys
d = snapshot_download("ThakiCloud/DocFields-F32-KO"); sys.path.insert(0, d)
import docfields_f32 as df
model, tok, labels, dec = df.load(d, device="cpu")
boxes = [{"text": "계좌번호", "bbox": [0.44, 0.31, 0.56, 0.34]},
{"text": "000-00-000000", "bbox": [0.60, 0.31, 0.84, 0.34]}]
for r in df.predict(model, tok, labels, dec, boxes):
print(r["text"], "->", r["label"])
Things you must keep the same as training, all handled by docfields_f32.py:
- One box is one unit. Only the first sub-token of each box carries the prediction; at most 24 sub-tokens per box.
- Pages longer than 510 sub-tokens are split on box boundaries, without overlap.
- Decoding is not plain argmax.
decode.jsonholds{"tau_date": null, "tau_acct": 0.25}: a box whose argmax is not a date or account label is promoted toACCOUNT_NOwhen p(ACCOUNT_NO) ≥ 0.25. Date promotion is off. The numbers below were measured with this decoding. Plainargmaxor the stockpipeline()gives different results. - Reading order matters to the encoder. Feed boxes top-to-bottom, left-to-right, as your OCR emits them.
What the shipped example returns
examples/synthetic_form.json is a hand-written 19-box claim form. On CPU the released weights label it as follows
(field keys such as "성명" come out as KEY):
| Box text | Label |
|---|---|
| 010-0000-0000 | PHONE |
| 서울특별시 가나구 다라로 123, 4층 | ADDRESS |
| 1,250,000원 | CLAIM_AMOUNT |
| 가나은행 | BANK_NAME |
| 000-00-000000 | ACCOUNT_NO |
| 2025년 3월 10일 (signing date) | ISSUE_DATE |
| 2025. 3. 4. (the value next to "사고일자", accident date) | ISSUE_DATE — arguably should be OTHER_DATE |
| 홍길동 (applicant name) | O — missed |
One easy form is a smoke test, not evidence of accuracy. It does show two typical weak spots: issue date versus other dates, and labels that were learned from synthetic forms only.
Labels (15)
O, KEY, APPLICANT_NAME, RRN_OR_BIRTH, ACCOUNT_NO, BANK_NAME, CONTRACT_NO, INS_PERIOD, CLAIM_AMOUNT,
CONSENT_MARK, SIGNATURE, ISSUE_DATE, PHONE, ADDRESS, OTHER_DATE
Only six of these were trained and scored on real forms: ADDRESS, PHONE, ACCOUNT_NO, CLAIM_AMOUNT,
ISSUE_DATE, OTHER_DATE. The others (APPLICANT_NAME, RRN_OR_BIRTH, BANK_NAME, CONTRACT_NO, INS_PERIOD,
SIGNATURE, and nearly all of KEY) were learned from synthetic forms only, and CONSENT_MARK had no training
examples at all. Treat those outputs as unverified.
1. What was measured
All numbers come from one evaluation set, called sealed7_fin in our ledger: 204 financial-form documents
(198 with complete labels). Read section 2 before using any of them.
| Metric (sealed7_fin) | F3.2 (this model) | Previous single model (F30m) | Previous 3-model routing (F3.1) | Target | Met |
|---|---|---|---|---|---|
| Macro-F1 over 5 roles | 0.9035 | 0.879 | 0.890 | ≥ 0.890 | yes |
| Date detection recall | 0.703 | 0.273 | 0.606 | ≥ 0.606 | yes |
| Date detection precision | 0.417 | 0.417 | 0.337 | ≥ 0.337 | yes |
ACCOUNT_NO F1 |
0.797 | 0.694 | 0.767 | ≥ 0.767 | yes |
CLAIM_AMOUNT F1 |
0.9231 | 0.9231 | – | ≥ 0.9181 | yes |
ADDRESS F1 |
0.9978 | 0.9985 | – | ≥ 0.9935 | yes |
PHONE F1 |
0.9147 | 0.9219 | – | ≥ 0.9169 | no |
ISSUE_DATE F1 |
0.885 | 0.857 | – | – | – |
- The five roles in macro-F1 are account number, address, claim amount, issue date and phone.
- "Date detection" asks whether a box holding a complete date string gets any date label (
ISSUE_DATEorOTHER_DATE). The reference is a deterministic date-format checker, not a human. Precision of 0.417 means more than half of the boxes the model calls a date are not complete dates by that checker: 0.79 false date boxes per document. - The phone target was the previous model's score minus 0.005. F3.2 lost 0.007. Every phone error we inspected was a
confusion between
PHONEandACCOUNT_NO. That is consistent with the account-number recovery pulling some phone numbers along, but we did not isolate the cause.
Paired bootstrap over documents (2,000 resamples, 95% interval):
| Difference | vs previous single model (F30m) | vs 3-model routing (F3.1) |
|---|---|---|
| Date detection recall | +0.430 [0.329, 0.534] | +0.097 [0.018, 0.181] |
| Date detection precision | – | +0.081 [0.044, 0.121] |
ACCOUNT_NO F1 |
+0.104 [0.016, 0.219] | +0.030 [−0.016, 0.087] |
| Macro-F1 | – | +0.014 [−0.016, 0.042] |
Against the three-model routing, macro-F1 and account F1 intervals include zero. We do not claim F3.2 is better than the routing on those; we claim it reaches a similar level with one checkpoint. All rows were scored by us with the same code and the same files.
Development gate that failed
Checkpoint selection used a separate development split (823 documents) with seven gates. The final checkpoint passed
six. It failed the claim-amount gate (0.8946 against a floor of 0.9013). It is the last-round checkpoint, evaluated
once as the pre-registration required, and reported as is. On that development split it scored macro-F1 0.858, date
recall 0.691 and date precision 0.166 — noticeably lower precision than on sealed7_fin.
On held-out synthetic forms F3.2 scored 0.843 against 0.844 for the previous model, i.e. no change.
2. Limits — observed improvement is not proven generalization
Please read the numbers above with all of the following in mind.
- The gold labels are pseudo-labels. They were produced by a large language model (an internal Qwen-family deployment we call "marm"), not by human annotators. The scores measure agreement with that labeler.
sealed7_finis an exposed evaluation set. It was used to evaluate earlier versions (F2.7, F3.0, F3.1) and was opened twice during F3.2 work. It is not a fresh test set and cannot support a release decision.- The initial weights had seen evaluation groups. F3.2 was initialized from F30m, whose training data overlapped 259 form groups (same template family) with the evaluation splits. F3.2's own training documents were filtered to zero id and zero group overlap, but exposure through the initial weights is not removed.
- Date detection recall is not an independent check. The same deterministic date checker was used as a condition when building training labels.
- No external evaluation exists. We have no independent set of real customer forms with human labels (E3 in our
ledger is
DATA_UNAVAILABLE). Accuracy on your forms is unknown. - Synthetic-form scores are not real-form scores, and are reported separately for that reason.
- Inputs are OCR boxes. OCR errors, different box granularity (line versus word), rotated or multi-column pages and forms outside insurance/banking paperwork were not measured.
- Numbers were measured on GPU with bf16 autocast. CPU fp32 inference, which the script defaults to, can differ in the last decimals.
What this release does not claim: readiness for production, accuracy on real forms, generalization to unseen templates or issuers, or superiority over the three-model routing on metrics whose interval includes zero.
3. How it was trained
- Base:
nielsr/lilt-xlm-roberta-base, full fine-tune, 15 labels. Initialized from our previous checkpoint (F30m); the 14 existing head rows were copied and only the newOTHER_DATErow was freshly initialized. OTHER_DATElabel restored. The predecessor had one date label (issue date) and pushed every other date toO, which taught the model to ignore dates. F3.2 gives non-issue dates their own label.- Uncertain boxes are masked out of the loss instead of being forced to
O: boxes where the pseudo-label and the date checker disagree, and account-shaped strings the pseudo-labeler did not mark. About 1,200 boxes were masked this way (another 3,147 carried a pseudo-role that has no label in this head and were masked as before). - Date/account discrimination. Account-shaped strings next to a contract-number, resident-number or phone key are
kept as hard negatives for
ACCOUNT_NO. - Distillation arm (the released one). For masked boxes, labels from earlier specialist models were used only when they passed a confidence condition; confirmed boxes were never overwritten. No new LLM labeling was done.
- Data: 1,059 real financial-form documents (567 more were excluded because their form group appeared in an evaluation split) replayed twice, plus 4,121 synthetic forms. AdamW, lr 2e-5, batch 8, 3 epochs, warmup 6% then linear decay, bf16. Two arms × two seeds × two rounds; this is arm B, seed 29, epoch 1 of round 2, chosen on the development split by a rule fixed before training.
4. Data and license
- Weights and code: MIT. Base model
nielsr/lilt-xlm-roberta-base: MIT. - Training data source: AI 허브(AI-Hub) — OCR 데이터(금융 및 물류) (71301), and Korean statutory form templates (law.go.kr) filled with synthetic values.
- The data is not redistributed. The AI-Hub dataset is for the approved applicant's internal use only and may not be redistributed. This repository contains weights, tokenizer, label names, the decoding threshold, code and one hand-written synthetic example — no dataset image, text, label, crop, subset or sample.
- Training forms contain personal-information fields. The model predicts field roles; do not use it to harvest personal data without a lawful basis.
5. Files
model.safetensors (sha256 2bc3269f764d248666414608dda10d4c836336e2eaadb289db5de98c54829f9f), config.json,
tokenizer.json, tokenizer_config.json, special_tokens_map.json, labels.json, decode.json,
docfields_f32.py, requirements.txt, examples/synthetic_form.json, LICENSE, NOTICE.