Optimal Bandwidth Allocation for Web Crawler Systems

Web crawler is an important tool to obtain the information from the Internet in time. In a typical web crawler system with the limited bandwidth, there are many websites required to be crawled with different time constraints. Existing works about web crawler systems have not considered the bandwidth allocation in such a complex environment and hence the time constrains may not be properly satisfied. In this study, we investigate the bandwidth allocation approaches for such a web crawler system. The approaches are designed for both the scenarios that the sequences of crawling websites can not be changed and the scenarios that can. For the former scenario, we propose approaches to control bandwidth for web crawlers to minimize the maximum complete time or minimize the sum of complete times of all web crawlers. For the latter scenario, we propose a greedy algorithm based on crawling priorities, and a optimal algorithm based on mixed integer programming. Extensive simulations are conducted for validating the proposed approaches, and the result show that our approaches achieve desirable performance.