An Algorithm on Web Article Automatic Extraction Based on DOM Structure

By analyzing a large number of web pages, we proposed a page segment algorithm which is based on DOM tree structural features and visual features. The algorithm segments the pages to the small particles. It produces basic processing units for recognition algorithms. After segmenting the pages, we extracted the structural and visual features of the pages, and proposed a method to identify the body of the web article. The method uses clustering algorithm and heuristic rules to produce an automatic wrapper. A testing experiment demonstrated the efficacy of the algorithm.