FFDetr, ONNX at 1024x1024

jbarrow/FFDetr converted to ONNX. FFDetr is Joe Barrow's form field detector, an RF-DETR Medium model trained on the CommonForms dataset. Given an image of a form page, it finds the text boxes, checkboxes and signature fields.

DocSpring uses this file to detect the fields on uploaded PDFs. We publish it so our builds can download one pinned file instead of exporting it from PyTorch each time, and so anyone else who wants to run FFDetr without PyTorch can use it too.

What the conversion changed

The weights are FFDetr's, unchanged. The export differs from rfdetr's defaults in two ways:

• Input size 1024x1024. The checkpoint was trained at 1024x1024 (its saved training arguments say so), but rfdetr's RFDETRMedium runs at 576 by default. At 1024 it finds noticeably more fields (see below).
• Position embeddings stored as a constant. At any size but 576, the DINOv2 backbone resizes its position embeddings with antialiased bicubic interpolation, which ONNX can't express. The result depends only on the input size, so it was computed once in PyTorch and stored in the graph.

ONNX Runtime gives the same boxes as the PyTorch model.

Exported from FFDetr.pth at revision 56f4e4235e28dcb2953513dc020bb191a2f54cfe of jbarrow/FFDetr, with rfdetr 1.6.5.post0 and torch 2.11.0, ONNX opset 17.

Input and output

• input: float32 [1, 3, 1024, 1024]. The page as RGB scaled to 0-1, resized to 1024x1024 with an antialiased bilinear filter (as torchvision and Pillow do it), then normalised with the ImageNet mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225].
• dets: float32 [1, 300, 4]. One box per query as centre x, centre y, width and height, from 0 to 1 relative to the page.
• labels: float32 [1, 300, 91]. Class logits. FFDetr uses the first three classes: 0 text box, 1 choice button (a checkbox or radio button), 2 signature.

To get boxes the way rfdetr does: apply a sigmoid to the logits, take the 300 best query and class scores, keep the ones above a confidence threshold, scale the boxes to the page and remove overlaps with class-agnostic NMS. DocSpring uses a confidence of 0.3 and an NMS IoU of 0.1.

Accuracy

On the 164 pages of the CommonForms test split (5,294 fields), counting a field as found when a box overlaps it at IoU 0.3 or more:

Licence and credit

Apache-2.0, the same as FFDetr. The model and the CommonForms dataset are by Joe Barrow: CommonForms: A Large, Diverse Dataset for Form Field Detection. FFDetr was trained with Roboflow's RF-DETR (Apache-2.0). DocSpring converted the file to ONNX as described above.