Description: Cola is a distributed crawler frame, users only need to write a few specific functions, without attention to detail distributed operation. Tasks are automatically assigned to multiple machines, the entire process is transparent to users.
To Search:
File list (Check if you may need any files):
cola-master
...........\.gitignore
...........\AUTHORS
...........\LICENSE
...........\MANIFEST.in
...........\README.rst
...........\app
...........\...\__init__.py
...........\...\weibo
...........\...\.....\__init__.py
...........\...\.....\bundle.py
...........\...\.....\conf.py
...........\...\.....\login.py
...........\...\.....\parsers.py
...........\...\.....\requirements.txt
...........\...\.....\storage.py
...........\...\.....\utils.py
...........\...\.....\weibo.yaml
...........\...\wiki
...........\...\....\__init__.py
...........\...\....\requirements.txt
...........\...\....\wiki.yaml
...........\cola
...........\....\__init__.py
...........\....\cluster
...........\....\.......\__init__.py
...........\....\.......\master.py
...........\....\.......\stage.py
...........\....\.......\tracker.py
...........\....\.......\worker.py
...........\....\cmdline.py
...........\....\commands
...........\....\........\__init__.py
...........\....\........\job.py
...........\....\........\master.py
...........\....\........\startproject.py
...........\....\........\worker.py
...........\....\conf
...........\....\....\main.yaml
...........\....\context.py
...........\....\core
...........\....\....\__init__.py
...........\....\....\bloomfilter
...........\....\....\...........\__init__.py
...........\....\....\...........\hashtype.py
...........\....\....\config.py
...........\....\....\counter.py
...........\....\....\dedup.py
...........\....\....\errors.py
...........\....\....\extractor
...........\....\....\.........\__init__.py
...........\....\....\.........\preprocess.py
...........\....\....\.........\readability.py
...........\....\....\.........\utils.py
...........\....\....\handlers.py
...........\....\....\logs.py
...........\....\....\mq
...........\....\....\..\__init__.py
...........\....\....\..\client.py
...........\....\....\..\distributor.py
...........\....\....\..\hash_ring.py
...........\....\....\..\node.py
...........\....\....\..\store.py
...........\....\....\..\utils.py
...........\....\....\opener.py
...........\....\....\parsers.py
...........\....\....\rpc.py
...........\....\....\unit.py
...........\....\....\urls.py
...........\....\....\utils.py
...........\....\....\zip.py
...........\....\functions
...........\....\.........\__init__.py
...........\....\.........\budget.py
...........\....\.........\counter.py
...........\....\.........\speed.py
...........\....\job
...........\....\...\__init__.py
...........\....\...\container.py
...........\....\...\executor.py
...........\....\...\task.py
...........\....\settings.py
...........\....\templates
...........\....\.........\project.py.tmpl
...........\....\.........\project.yaml.tmpl
...........\lab
...........\...\generic
...........\...\.......\__init__.py
...........\...\.......\generic.yaml
...........\...\weibosearch
...........\...\...........\__init__.py
...........\...\...........\bundle.py
...........\...\...........\conf.py
...........\...\...........\keywords.txt
...........\...\...........\login.py
...........\...\...........\parsers.py
...........\...\...........\starts.py
...........\...\...........\storage.py
...........\...\...........\weibosearch.yaml
...........\requirements.txt
Download