Description: Based WebCollector kernel, you can write your own http request reptiles, link resolver, crawling updates, spider and other modules, WebCollector write these kernel-based modules called plug-in through a combination of different plug-ins, you can within one minute, the WebCollector assembled into a new reptile.
WebCollector built a set of plug-ins (cn.edu.hfut.dmic.webcollector.plugin.redis). Based on this set of plug-ins, you can put WebCollector task management onto the redis , which makes WebCollector be crawling mass of data (on a million levels).
Users can according to their own needs, to develop their own plug-ins. .
To Search:
File list (Check if you may need any files):
WebCollector-master
...................\.gitignore
...................\LICENSE
...................\README.md
...................\README.zh-cn.md
...................\WebCollector
...................\............\pom.xml
...................\............\src
...................\............\...\main
...................\............\...\....\java
...................\............\...\....\....\cn
...................\............\...\....\....\..\edu
...................\............\...\....\....\..\...\hfut
...................\............\...\....\....\..\...\....\dmic
...................\............\...\....\....\..\...\....\....\webcollector
...................\............\...\....\....\..\...\....\....\............\crawler
...................\............\...\....\....\..\...\....\....\............\.......\BreadthCrawler.java
...................\............\...\....\....\..\...\....\....\............\fetcher
...................\............\...\....\....\..\...\....\....\............\.......\Fetcher.java
...................\............\...\....\....\..\...\....\....\............\generator
...................\............\...\....\....\..\...\....\....\............\.........\DbUpdater.java
...................\............\...\....\....\..\...\....\....\............\.........\Generator.java
...................\............\...\....\....\..\...\....\....\............\.........\Injector.java
...................\............\...\....\....\..\...\....\....\............\.........\ParamURLGenerator.java
...................\............\...\....\....\..\...\....\....\............\.........\StandardGenerator.java
...................\............\...\....\....\..\...\....\....\............\.........\filter
...................\............\...\....\....\..\...\....\....\............\.........\......\Filter.java
...................\............\...\....\....\..\...\....\....\............\.........\......\IntervalFilter.java
...................\............\...\....\....\..\...\....\....\............\.........\......\URLRegexFilter.java
...................\............\...\....\....\..\...\....\....\............\.........\......\UniqueFilter.java
...................\............\...\....\....\..\...\....\....\............\handler
...................\............\...\....\....\..\...\....\....\............\.......\Handler.java
...................\............\...\....\....\..\...\....\....\............\.......\Message.java
...................\............\...\....\....\..\...\....\....\............\model
...................\............\...\....\....\..\...\....\....\............\.....\AvroModel.java
...................\............\...\....\....\..\...\....\....\............\.....\CommonURL.java
...................\............\...\....\....\..\...\....\....\............\.....\CrawlDatum.java
...................\............\...\....\....\..\...\....\....\............\.....\Link.java
...................\............\...\....\....\..\...\....\....\............\.....\Page.java
...................\............\...\....\....\..\...\....\....\............\output
...................\............\...\....\....\..\...\....\....\............\......\FileSystemOutput.java
...................\............\...\....\....\..\...\....\....\............\parser
...................\............\...\....\....\..\...\....\....\............\......\HtmlParser.java
...................\............\...\....\....\..\...\....\....\............\......\LinkUtils.java
...................\............\...\....\....\..\...\....\....\............\......\ParseResult.java
...................\............\...\....\....\..\...\....\....\............\......\Parser.java
...................\............\...\....\....\..\...\....\....\............\ui
...................\............\...\....\....\..\...\....\....\............\..\BreadthCrawlerUI.form
...................\............\...\....\....\..\...\....\....\............\..\BreadthCrawlerUI.java
...................\............\...\....\....\..\...\....\....\............\util
...................\............\...\
Download