Description: NwebCrawler is a multi-threaded web crawler program written in C#, it is the realization of the principle of the first to enter one or more seed URL to the queue, and then extracted from the queue URL (FIFO principle), analysis of this page to find the appropriate label and get the href attribute value, useful links page crawling and deposited in the web library, with crawling history to record climbed pages, so to avoid a repeat of crawling. Extract URL stored in the queue, and the next one crawling. NwebCrawler search strategy is breadth-first search. Breadth-first strategy is conducive to multiple threads parallel crawling and crawling closed strong.
To Search:
File list (Check if you may need any files):
源代码
......\CrawlerThread.cs
......\CrawlHistroyEntry.cs
......\Downloader.cs
......\EventStream.cs
......\MemCache.cs
......\NWebCrawlerLib.csproj
......\Parser.cs
......\Program.cs
......\Settings.cs
......\UrlQueueManager.cs
......\Utility.cs
Download