[磋商]乡村产业全过程咨询项目其他农业服务采购项目01包：乡村振兴产业综合发展项目全过程技术咨询、03包：京郊休闲农业和乡村旅游“百园百线”线上线下推介活动、04包：京蒙协作资金项目前期评估成交公告

0211202/Python-Python-Scrapy-学习笔记

# Scrapy学习笔记

1. Scrapy安装

• pip install scrapy

2. Scrapy基本使用

1. 创建一个项目：

```
$ scrapy startproject myproject

```
1. 进入项目目录：

```
$ cd myproject

```
1. 创建一个spider：

```
$ scrapy genspider example com

```

参数说明：

• example: spider的名字
• com: 爬虫抓取的目标网站的域名后缀（例如example.com）
• 可以使用默认的爬虫名字和目标域名后缀（`$ scrapy genspider -d example.com`）
• 如果不指定爬虫名字和目标域名后缀，那么会输出一个默认的爬虫名字和目标域名后缀（`$ scrapy genspider`）

```
$ scrapy genspider example example.com

```

```
$ scrapy genspider -d example.com

```
1. 编写爬虫：

```
# myproject/spiders/example.py

import scrapy
from myproject.items import MyprojectItem

class ExampleSpider(scrapy.Spider):
name = 'example'
allowed_domains = ['example.com']
start_urls = ['http://example.com/']

def parse(self, response):
item = MyprojectItem()
item['title'] = response.xpath('//title/text()').extract_first()
item['content'] = response.xpath('//div[@id="content"]/p/text()').extract()
return item

```
1. 启动爬虫：

```
$ scrapy crawl example

```

```
$ scrapy crawl -s example

```
1. 数据处理：

1. 保存爬取的数据：

```
$ scrapy crawl example -o output.json

```

```
$ scrapy crawl example -o output.csv

```
1. 输出到数据库：

```
$ scrapy crawl example -o output.sql

```
1. 输出到文件夹：

```
$ scrapy crawl example -o output/

```

```
$ scrapy crawl example -o output/ -t json

```

```
$ scrapy crawl example -o output/ -t xml

```

```
$ scrapy crawl example -o output/ -t csv

```
1. 输出到MongoDB：

```
$ scrapy crawl example -o output/ -t mongodb

```
1. 输出到Redis：

```
$ scrapy crawl example -o output/ -t redis

```

3. Scrapy配置

3.1 settings.py

1. 编辑settings.py配置文件：

```
# myproject/settings.py

import os
os.environ.get('DJANGO_SETTINGS_MODULE', 'myproject.settings')

```

```
# myproject/settings.py

BOT_NAME = 'example'

SPIDER_MODULES = ['myproject.spiders']
NEWSPIDER_MODULE = 'myproject.spiders'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 16

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings to limit download rate
DOWNLOAD_DELAY = 1

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36'

```

3.2 环境变量

• 为了在settings.py配置文件中设置环境变量，首先需要在系统中设置环境变量。

• 环境变量设置方法：

1. 设置环境变量：

```
$ export DJANGO_SETTINGS_MODULE=myproject.settings

```

```
$ set DJANGO_SETTINGS_MODULE=myproject.settings

```
1. 测试环境变量设置是否成功：

```
$ env | grep DJANGO

```

```
$ echo $DJANGO_SETTINGS_MODULE

```

```
$ echo $DJANGO_SETTINGS_MODULE

myproject.settings

```
1. 保存环境变量到本地配置文件：

```
$ export > .profile

```

```
$ set > .bashrc

```

```
$ export > .bash_profile

```

```
$ set > .bashrc

```

```
$ export > .bashrc

```
• 退出环境变量设置：

```
$ unset DJANGO_SETTINGS_MODULE

```

```
$ unset DJANGO_SETTINGS_MODULE

```

```
$ unset $DJANGO_SETTINGS_MODULE

```

```
$ unset $DJANGO_SETTINGS_MODULE

```
1. 清除环境变量：

```
$ unset

```

```
$ unset

```

```
$ unset -v

```

```
$ unset -v

```

```
$ unset -f

```

```
$ unset -f

```

4. Scrapy框架

• Scrapy框架是由Scrapy项目组开发的Python网页抓取框架。它具有以下优点：

• 可复用性强