Combining a Segmentation-Like Approach and a Density-Based Approach in Content Extraction

Density-based approaches in content extraction, whose task is to extract contents from Web pages, are commonly used to obtain page contents that are critical to many Web mining applications. How- ever, traditional density-based approaches cannot effectively manage pages that contain short contents and long noises. To overcome this problem, in this paper, we propose a content extraction approach for obtain- ing content from news pages that combines a segmentation-like approach and a density-based approach. A tool called BlockExtractor was developed based on this approach. BlockExtractor identifies contents in three steps. First, it looks for all Block-Level Elements （BLE） ＆ Inline Elements （IE） blocks, which are designed to roughly segment pages into blocks. Second, it computes the densities of each BLE＆IE block and its ele- ment to eliminate noises. Third, it removes all redundant BLE＆IE blocks that have emerged in other pages from the same site. Compared with three other density-based approaches, our approach shows significant advantages in both precision and recall.