GitHub

工商信息离线抓取系统

总入口文件: start_task_crawler.py

同步种子脚本: start_update_process.py

db.getCollection('online_all_search').update({province:'gansu'},{$unset:{'crawl_online':''}},false, true)

Name		Name	Last commit message	Last commit date
Latest commit History 14 Commits
base		base
bdp		bdp
bin		bin
check		check
common		common
config		config
doc		doc
downloader		downloader
i_util		i_util
task		task
tests		tests
thriftRPC		thriftRPC
tools		tools
.gitignore		.gitignore
README.md		README.md
__init__. 8000 py		__init__.py
env.sh		env.sh
logger.py		logger.py
requirements.txt		requirements.txt
start_offline_all_process.sh		start_offline_all_process.sh
start_offline_crawler.py		start_offline_crawler.py
start_offline_detail_process.sh		start_offline_detail_process.sh
start_offline_searchlist_process.sh		start_offline_searchlist_process.sh
start_online_crawl.py		start_online_crawl.py
start_online_crawl_process.sh		start_online_crawl_process.sh
update_offline_all_process.sh		update_offline_all_process.sh
update_offline_process.sh		update_offline_process.sh
update_online_process.sh		update_online_process.sh

Provide feedback