Introduction - If you have any usage issues, please Google them yourself
Web crawlers Download Web pages from the world wide web for search engines. Generally divided into traditional reptiles and focused crawler.
The traditional crawler starts from one or several "initial URL, the initial URL on the page, in the process of crawling, continuously from the current page from the new URL queue, until the system must stop condition. Popular speaking, that is, through the source code to get the content you want.