A Novel Distributed Web Crawling Approach Based on Mongodb

The crawler is an important part of Web Search Engine, but most crawlers, as a single-point technology, were often limited by the hardware to create value. How to build an efficient, stable, and scalable distributed crawling system has become an important issue. A distributed crawling technology combined Mongodb is described in this paper and it is also showed that how the technology is used in practice to build a small-scale distributed system. Finally, this technology is more efficient, stable and easier to expand than others as compared with single-point crawling technology and similar products of NoSQL.