Introduction - If you have any usage issues, please Google them yourself
heritrix is an open source network reptiles/Web Spiders, heritrix purpose is to track the page url to the expansion of the crawl, and finally for the search engine provides a wide range of data sources.