scrapy抓取动态页面 scrapy抓取动态页面

一、准备工作
1.安装splash 在windows环境下，splash可通过docker进行安装，安装方法在这篇文章中有详细讲解，在此不再赘述。
安装完成后在powershell中运行以下命令安装splash：

#从docker hub下载相关镜像文件 docker pull scrapinghub/splash、

使用docker启动服务命令启动Splash服务，在8050、8051、5023端口开启splash服务：

#启动splash服务，并通过http，https，telnet提供服务 #通常一般使用http模式，可以只启动一个8050就好 #Splash 将运行在 0.0.0.0 at ports 8050 (http), 8051 (https) and 5023 (telnet). docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash

在浏览器中输入10.0.75.1:8050查看服务启动情况：

文章图片
QQ截图20180214144552.png 2.安装scrapy-splash 使用pip安装scrapy-splash：

pip install scrapy-splash

完成以上步骤之后就可以在scrapy中使用splash了
二、编写爬虫
动态页面中的部分内容是浏览器运行页面中的javascript脚本生成的，相较于普通静态页面，爬取难度更大。以网站 http://quotes.toscrape.com/js/为例，爬取动态生成的名人名言信息。
在项目文件settings.py中配置scrapy-splash：

#定义splash服务器地址 SPLASH_URL = 'http://10.0.75.1:8050/' #开启splash的下载中间件 DOWNLOADER_MIDDLEWARES = { 'scrapy_splash.SplashCookiesMiddleware': 723, 'scrapy_splash.SplashMiddleware': 725, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810, } #设置去重 DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

编码实现：

# -*- coding: utf-8 -*- import scrapy from scrapy_splash import SplashRequestclass QuoteSpider(scrapy.Spider): name = 'quote' start_urls = ['http://quotes.toscrape.com/js/']def start_requests(self): for url in self.start_urls: yield SplashRequest(url, args={'images': 0})def parse(self, response): quotes = response.xpath('//div[@class="quote"]') for quote in quotes: text = quote.xpath('.//*[@class="text"]/text()').extract_first() author = quote.xpath('.//*[@class="author"]/text()').extract_first() yield { 'text': text, 'author': author, } next_page = response.xpath('//a[contains(text(),"Next")]/@href').extract_first() if next_page: absolute_url = response.urljoin(next_page) yield SplashRequest(absolute_url, args={'images': 0})

执行结果：

2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/> {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein'} 2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/> {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling'} ....2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/> {'text': "“A person's a person, no matter how small.”", 'author': 'Dr. Seuss'} 2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/> {'text': '“... a mind needs books as a sword needs a whetstone, if it is to keep its edge.”', 'author': 'George R.R. Martin'}

【scrapy抓取动态页面】结果显示我们已成功爬取动态生成的名人名言信息。