NuExtract3-kie-vi

LoRA fine-tune of numind/NuExtract3 for key information extraction (KIE) on Vietnamese public-sector asset/income declaration forms ("Bản kê khai tài sản, thu nhập"), merged into full dense weights (no adapter needed to load).

Model Details

• Base model: numind/NuExtract3 — a Qwen3.5-4B vision-language model (Qwen3_5ForConditionalGeneration, hybrid full/linear attention) specialized for document → structured-JSON extraction. Apache-2.0.
• Fine-tuning method: LoRA, merged into base weights via peft's merge_and_unload(). This repo ships the merged checkpoint — load it like any ordinary dense model, no PEFT/adapter loading required.
  • r=32, alpha=64, dropout=0.05
  • Target modules: attention (q_proj/k_proj/v_proj/o_proj), MLP (gate_proj/up_proj/down_proj), and the vision tower's linear-attention projections (in_proj_a/in_proj_b/in_proj_qkv/in_proj_z, linear_fc1/linear_fc2) — i.e. LoRA was applied to the vision tower as well as the language model.
  • Trained with reasoning off (plain SFT on image→JSON pairs, no thinking trace).
• Task format: NuExtract3's native template-based structured extraction — input is a JSON template (field names + types) plus one or more page images of a declaration; output is JSON matching the template, mostly verbatim-string fields (values copied as printed, to avoid hallucinated reformatting) plus a handful of typed fields.
• Training data: 151 real, pre-labeled multi-page declaration documents (151 train / 19 val), used as full documents only (the source corpus has no word/box-level annotations, so no page- or section-level training samples were derived from it — see scripts/prepare_labeled_pdf_dataset.py). 3 epochs, effective batch size 16 (batch_size=1 × grad_accum=16), LR 1e-4 cosine, warmup 3%.

Evaluation status

Not yet formally evaluated on this checkpoint. The project's acceptance target is macro field accuracy ≥ 93% and field pass-rate > 90% (measured with kie.eval.run: precision/recall/F1, BLEU, ROUGE-L, plus per-field accuracy). Those numbers exist for other training configs in this project (a synthetic-data pilot run, and zero-shot baselines) but have not been re-run against this specific remote_lora checkpoint — do not assume it meets the target. Run the project's eval harness against your own gold set before relying on this checkpoint in production.

Known limitations (inherited from the base model + not yet addressed by this fine-tune)

• Vietnamese: NuExtract3's specialization layer was trained primarily on English/European documents (Fine-PDF); Vietnamese diacritics and layouts are not proven in the base model's own reporting. This LoRA fine-tune targets Vietnamese declarations but was trained on a small real-document set (151 docs) — treat as an early checkpoint, not a fully validated one.
• Handwriting: neither the base model nor this fine-tune has been specifically hardened for handwritten numeric fields (ID numbers, currency amounts, account numbers) — these are the highest-risk fields for silent errors.
• No abstention / "illegible" mechanism: the model will attempt to fill every templated field; it does not flag low-confidence or unreadable values. A downstream guardrail (cross-checking with OCR, confidence/consensus scoring, or an explicit ILLEGIBLE schema slot) is recommended before using extracted values for anything consequential.

Usage

Same interface as the base model — build the NuExtract3 template/instructions prompt via its chat template, pass page image(s), and parse the JSON output. See numind/NuExtract3 for the prompt format, or the NuExtract fine-tuning cookbook for reference code.

Because this checkpoint is a merged dense model, it serves directly on standard multimodal-LLM serving stacks (vLLM, SGLang, transformers) with no LoRA-specific flags needed.

Intended use

Internal document-extraction pipeline for Vietnamese public-sector asset/income declarations. Not intended as a general-purpose Vietnamese OCR/VLM model.