PESNet: Point-Edge-Semantics Building Extraction and Vectorization in Remote Sensing Images

When humans delineate objects of interest from high resolution remote sensing (RS) images, they first determine the existence and locations of these objects. Then, they identify the boundaries that distinguish them from other objects and finally fit the object boundaries by sketching key points of the object. However, existing methods often prioritize semantic information over edges, key points, and failing to meet the requirement for simplicity of results. To address this problem, we propose a novel network called PESNet (Point-Edge-Semantic) for extracting building key points, edges, and semantics results from RS images and combining them to produce vectorization results. PESNet utilizes a multi-tasking learning framework that incorporates three sub-tasks (semantic, edge, and key point). The key points are then connected based on the guidance of the edges and semantics. The resulting closed edges form the object polygons in vector format. To evaluate the performance of PESNet, we conducted experiments on two benchmark datasets. Extensive experiments on the ISPRS and CrowdAI datasets demonstrate that our proposed PESNet performs best in baseline methods, with IoU scores reaching 91.01% and 82.41%, respectively, outperforming the second-best baseline by 1.58% and 2.37%. Additionally, PESNet exhibits exceptional performance in edge extraction, surpassing the second-best baseline by 5.17% and 4.47%. Moreover, the vectorized building boundaries generated by PESNet exhibit regularity and simplicity.