scrapyd监控scrapy爬虫进度

发表于 | 分类于
字数统计 178 | 阅读时长 1

准备工作

安装scrapyd

1
pip install scrapyd

安装scrapyd-client

1
pip install scrapyd-client

运行scrapyd服务

1
scrapyd

Spider进度查看地址：http://localhost:6800/

20190905235045.png

部署scrapy项目

部署scrapy项目（修改Spider之后需要重新部署哦）

修改scrapy.cfg文件

1
2
3
4
5
6
7
8
[deploy] section see:

[settings]
default = demo.settings

[deploy:demo_deploy]
url = http://localhost:6800/
project = demo

直接在项目demo的根目录路运行部署命令：

1
2
3
4
5
➜  demo git:(master) ✗ scrapyd-deploy demo_deploy -p demo
Packing version 1513956720
Deploying to project "demo" in http://localhost:6800/addversion.json
Server response (200):
{"status": "ok", "project": "demo", "version": "1513956720", "spiders": 2, "node_name": "MacBook-Pro-4.local"}

添加Spider到任务队列

查看已有爬虫：

1
2
3
➜  demo git:(master) scrapy list
demo4399pk
yh31

添加Spider到任务队列：

1
curl http://localhost:6800/schedule.json -d project=demo -d spider=yh31

来源：http://leunggeorge.github.io/

0%