With advances in Document AI technology and the emergence of sophisticated AI models. Application of Document AI technology to parse and extract value from engineering drawings has potential to streamline related processes.

This RT-DETR model has been fine-tuned to detect the bill of materials, title block and notes from the following drawing types:

1: Isometric drawings

2: Piping and Instrumentation drawings.

3: Datasheets

The model provides a confidence score for each detection that can be used as a filter. Detections with high confidence scores can minimize false positives. The detections can be cropped and send to multimodal AI models such as Qwen 2.5VL for extraction of text in json format.

The model can be fine-tuned further to improve accuracy and handle different layouts for the title block and bill of materials.

Model files are available in yolo and onnx format.

Inference using ultralytics library

from ultralytics import RTDETR
import cv2
import supervision as sv 

# --------------------------
model_file = 'replace with path to model file eng_dwg_v1.pt'

# Load a trained model from local path
model = RTDETR(model_file)

# Display model information (optional)
model.info()

image_path = 'path to source image'

# read src image
img = cv2.imread(image_path)

# perform inference
results = model.predict(img, imgsz=1024) #imgsz is set to 1024 as the model is finetuned with this size

# use the supervision library for parsing results and generating redline boxes
detections = sv.Detections.from_ultralytics(results[0])

#get a bounding box and label the annotator
bounding_box_annotator = sv.BoundingBoxAnnotator() 
label_annotator = sv.LabelAnnotator()

# generate labels for images
labels = [
    f"{class_name} {confidence:.2f}"
    for class_name, confidence
    in zip(detections['class_name'], detections.confidence)
]

# annotate images with labeled bounding boxes
annotated_image = bounding_box_annotator.annotate(
    scene=img.copy(),
    detections=detections
)
annotated_image = label_annotator.annotate(annotated_image, detections=detections, labels=labels)
# dummy counter for generated image names
count = 0
# write the annotated image
cv2.imwrite('redlined_' + str(count) + '.png', annotated_image)

# crop bounding boxes and save 
for xyxy in detections.xyxy:
    cropped_image = sv.crop_image(image=img, xyxy=xyxy)
    count = count + 1
    cv2.imwrite('bboxes_' + str(count) + '.png', cropped_image)

Engineering Drawings BOM, Title Block, extraction example 1

Extracted Bill of Materials (BOM)

Extracted Bill of Materials (BOM) converted to json

{
  "BILL OF MATERIALS": [
    {
      "ITEM": "1",
      "QTY": "1",
      "DESCRIPTION": "Shell - 0.375 Thk",
      "MATERIAL": "SA-414 Gr G"
    },
    {
      "ITEM": "2",
      "QTY": "2",
      "DESCRIPTION": "Head - 36.00 OD, SE 2:1, 2.00 SF, .313 Thk, .281 MAF",
      "MATERIAL": "SA-414 Gr G"
    },
    {
      "ITEM": "3",
      "QTY": "2",
      "DESCRIPTION": "Saddle - .38 Thk, Formed",
      "MATERIAL": "SA-36"
    },
    {
      "ITEM": "4",
      "QTY": "4",
      "DESCRIPTION": "Saddle Rib - .38 Thk x 4.00 x 19.25",
      "MATERIAL": "SA-36"
    },
    {
      "ITEM": "5",
      "QTY": "4",
      "DESCRIPTION": "Saddle Rib - .38 Thk x 4.00 x 12.25",
      "MATERIAL": "SA-36"
    },
    {
      "ITEM": "6",
      "QTY": "1",
      "DESCRIPTION": "Skirt - .188 Thk. x 24\" LG",
      "MATERIAL": "SA-36"
    },
    {
      "ITEM": "7",
      "QTY": "4",
      "DESCRIPTION": "Angle Clip - 4.00 x 4.00 x .38 Angle x 4.00 LG",
      "MATERIAL": "SA-36"
    }
  ]
}

Extracted Title Block

Extracted Title Block converted to json

{
  "Title": "Generic Vessel",
  "Size": "B",
  "Job ID": "PVE-6847",
  "Drawing No.": "PVE-6847-Generic Vessel",
  "Revision": "0",
  "Scale": "1:24",
  "Material": "See BOM",
  "Sheet": "1 OF 1"
}

Engineering Drawings BOM, Title Block, extraction example 2

Extracted Bill of Materials (BOM)

Extracted Bill of Materials (BOM) converted to json

{
  "BILL OF MATERIALS": [
    {
      "ITEM": "1",
      "QTY": "1",
      "DESCRIPTION": "Shell - 0.375 Thk",
      "MATERIAL": "SA-414 Gr G"
    },
    {
      "ITEM": "2",
      "QTY": "2",
      "DESCRIPTION": "Head - 36.00 OD, SE 2:1, 2.00 SF, .313 Thk, .281 MAF",
      "MATERIAL": "SA-414 Gr G"
    },
    {
      "ITEM": "3",
      "QTY": "2",
      "DESCRIPTION": "Saddle - .38 Thk, Formed",
      "MATERIAL": "SA-36"
    },
    {
      "ITEM": "4",
      "QTY": "4",
      "DESCRIPTION": "Saddle Rib - .38 Thk x 4.00 x 19.25",
      "MATERIAL": "SA-36"
    },
    {
      "ITEM": "5",
      "QTY": "4",
      "DESCRIPTION": "Saddle Rib - .38 Thk x 4.00 x 12.25",
      "MATERIAL": "SA-36"
    },
    {
      "ITEM": "6",
      "QTY": "1",
      "DESCRIPTION": "Full Coupling, B16.11 NPT - 3000#, 1.50 NPS",
      "MATERIAL": "SA-105"
    },
    {
      "ITEM": "7",
      "QTY": "1",
      "DESCRIPTION": "Flange, B16.5 RFSO - 150#, 4.00 NPS",
      "MATERIAL": "SA-105"
    },
    {
      "ITEM": "8",
      "QTY": "1",
      "DESCRIPTION": "Pipe - Sch. Std., 4.00 NPS",
      "MATERIAL": "SA-106 Gr. B"
    },
    {
      "ITEM": "9",
      "QTY": "3",
      "DESCRIPTION": "Full Coupling, B16.11 NPT - 3000#, 1.00 NPS",
      "MATERIAL": "SA-105"
    },
    {
      "ITEM": "10",
      "QTY": "1",
      "DESCRIPTION": "Pipe - Sch. Std., 6.00 NPS",
      "MATERIAL": "SA-106 Gr. B"
    },
    {
      "ITEM": "11",
      "QTY": "1",
      "DESCRIPTION": "Flange, B16.5 RFSO - 150#, 6.00 NPS",
      "MATERIAL": "SA-105"
    }
  ]
}

Extracted Title Block

Extracted Notes

Extracted Notes converted to json

{
 "NOTES": [
   {
     "item": "1.",
     "description": "All dimensions in inches."
   },
   {
     "item": "2.",
     "description": "All welds shall be neat in appearance, free from slag and other defects."
   },
   {
     "item": "3.",
     "description": "Vessel to be cleaned of scale, oil, weld spatter and all other foreign material, prior to hydrostatic test."
   },
   {
     "item": "4.",
     "description": "Remove all sharp edges on nozzles (1/8\" minimum radius)"
   },
   {
     "item": "5.",
     "description": "All nozzles to support nominal loads only."
   },
   {
     "item": "6.",
     "description": "Maximum misalignment of butt joints is limited to .25T (Category A, B, C, D up to 1/2\" thick)"
   },
   {
     "item": "7.",
     "description": "All couplings to conform to B16.11 (2011 Add.) standards."
   },
   {
     "item": "8.",
     "description": "All flanges to conform to B16.5 (2009 Add.)/ B16.47 standards."
   },
   {
     "item": "9.",
     "description": "Flange bolt holes to straddle natural centre lines."
   }
 ],
 "additional_info": {
   "drawing": "PVE-6847-Generic Vessel",
   "CRN information": "See drawing PVE-6847-Generic Vessel for CRN information"
 }
}