A task‐centric knowledge graph construction method based on multi‐modal representation learning for industrial maintenance automation

Maintenance manuals are crucial information sources for maintenance and repair. Prior studies explored factual knowledge extraction from textual documents. However, maintenance knowledge in manuals is more task‐centric rather than factual knowledge and often documented in an unstructured Portable Document Format (PDF), posing challenges for knowledge extraction. Addressing this, this research develops effective methods to extract task‐centric maintenance knowledge from unstructured PDF manuals. A new Task‐centric Knowledge Graph (TCKG) schema centralized on maintenance task components (MTCs) is proposed to address the need for structured knowledge representation. A method (Heterogeneous Graph‐based Method, HGM) for knowledge extraction is then proposed, which is enhanced by incorporating visual and spatial information. In the experiments, the proposed HGM exhibits robust performance in the knowledge extraction process, surpassing the baseline Graph‐based Interaction Model with a Tracker (GIT) method in MTCs extraction by 13.3%, and the baseline Translate Embedding (TransE) method in MTCs' relation extraction by 3.8%. A series of ablation studies also prove that including visual and spatial information through the proposed method can improve the relation extraction performance by over 10%. This research supplies valuable insights for future developments in information extraction from maintenance manuals. Our study introduces a new approach to extract maintenance knowledge from unstructured PDF manuals and construct task‐centric knowledge graph. This method significantly improves task and relation extraction, demonstrating a new paradigm in intelligent information processing in maintenance manuals.