0211202/Python-Python-Scrapy-学习笔记
# Scrapy学习笔记
1. Scrapy安装
• pip install scrapy
2. Scrapy基本使用
1. 创建一个项目：
```
$ scrapy startproject myproject
```
1. 进入项目目录：
```
$ cd myproject
```
1. 创建一个spider：
```
$ scrapy genspider example com
```
参数说明：
• example: spider的名字
• com: 爬虫抓取的目标网站的域名后缀（例如example.com）
• 可以使用默认的爬虫名字和目标域名后缀（`$ scrapy genspider -d example.com`）
• 如果不指定爬虫名字和目标域名后缀，那么会输出一个默认的爬虫名字和目标域名后缀（`$ scrapy genspider`）
```
$ scrapy genspider example example.com
```
```
$ scrapy genspider -d example.com
```
1. 编写爬虫：
```
# myproject/spiders/example.py
import scrapy
from myproject.items import MyprojectItem
class ExampleSpider(scrapy.Spider):
name = 'example'
allowed_domains = ['example.com']
start_urls = ['http://example.com/']
def parse(self, response):
item = MyprojectItem()
item['title'] = response.xpath('//title/text()').extract_first()
item['content'] = response.xpath('//div[@id="content"]/p/text()').extract()
return item
```
1. 启动爬虫：
```
$ scrapy crawl example
```
```
$ scrapy crawl -s example
```
1. 数据处理：
1. 保存爬取的数据：
```
$ scrapy crawl example -o output.json
```
```
$ scrapy crawl example -o output.csv
```
1. 输出到数据库：
```
$ scrapy crawl example -o output.sql
```
1. 输出到文件夹：
```
$ scrapy crawl example -o output/
```
```
$ scrapy crawl example -o output/ -t json
```
```
$ scrapy crawl example -o output/ -t xml
```
```
$ scrapy crawl example -o output/ -t csv
```
1. 输出到MongoDB：
```
$ scrapy crawl example -o output/ -t mongodb
```
1. 输出到Redis：
```
$ scrapy crawl example -o output/ -t redis
```
3. Scrapy配置
3.1 settings.py
1. 编辑settings.py配置文件：
```
# myproject/settings.py
import os
os.environ.get('DJANGO_SETTINGS_MODULE', 'myproject.settings')
```
```
# myproject/settings.py
BOT_NAME = 'example'
SPIDER_MODULES = ['myproject.spiders']
NEWSPIDER_MODULE = 'myproject.spiders'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 16
# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings to limit download rate
DOWNLOAD_DELAY = 1
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36'
```
3.2 环境变量
• 为了在settings.py配置文件中设置环境变量，首先需要在系统中设置环境变量。
• 环境变量设置方法：
1. 设置环境变量：
```
$ export DJANGO_SETTINGS_MODULE=myproject.settings
```
```
$ set DJANGO_SETTINGS_MODULE=myproject.settings
```
1. 测试环境变量设置是否成功：
```
$ env | grep DJANGO
```
```
$ echo $DJANGO_SETTINGS_MODULE
```
```
$ echo $DJANGO_SETTINGS_MODULE
myproject.settings
```
1. 保存环境变量到本地配置文件：
```
$ export > .profile
```
```
$ set > .bashrc
```
```
$ export > .bash_profile
```
```
$ set > .bashrc
```
```
$ export > .bashrc
```
• 退出环境变量设置：
```
$ unset DJANGO_SETTINGS_MODULE
```
```
$ unset DJANGO_SETTINGS_MODULE
```
```
$ unset $DJANGO_SETTINGS_MODULE
```
```
$ unset $DJANGO_SETTINGS_MODULE
```
1. 清除环境变量：
```
$ unset
```
```
$ unset
```
```
$ unset -v
```
```
$ unset -v
```
```
$ unset -f
```
```
$ unset -f
```
4. Scrapy框架
• Scrapy框架是由Scrapy项目组开发的Python网页抓取框架。它具有以下优点：
• 可复用性强