What is a crawling infrastructure?
12/06/2022 | By: FDS
A crawling infrastructure is a system that enables automatic crawling and indexing of web pages. It consists of a set of components that control, collect, and store the process of crawling. These include crawlers, indexers, repositories and managers. It is used to collect, store and index web documents for later use in searching for information.