Description: Internet Robot (Robot) also known as Spider, Worm, or Random, the core objective is to obtain information on the Internet. Home of the robot using hypertext links traversal Web, through the URL reference from a HTML document to another HTML document crawling. Online robot collected information can be a variety of purposes, such as indexing, HTML document verification, URL link verification, access to updated information, such as mirror sites.
File list (Check if you may need any files):
Robot.txt
用C#语言构造蜘蛛程序 .txt
Download