Python中scrapy下载保存图片的示例 Python中scrapy下载保存图片的示

在日常爬虫练习中，我们爬取到的数据需要进行保存操作，在scrapy中我们可以使用ImagesPipeline这个类来进行相关操作，这个类是scrapy已经封装好的了，我们直接拿来用即可。

文章图片

在使用ImagesPipeline下载图片数据时，我们需要对其中的三个管道类方法进行重写，其中— get_media_request是对图片地址发起请求
— file path是返回图片名称
— item_completed返回item,将其返回给下一个即将被执行的管道类

文章图片

那具体代码是什么样的呢，首先我们需要在pipelines.py文件中，导入ImagesPipeline类，然后重写上述所说的3个方法：

from scrapy.pipelines.images import ImagesPipelineimportscrapyimport os class ImgsPipLine(ImagesPipeline):def get_media_requests(self, item, info):yield scrapy.Request(url = item['img_src'],meta={'item':item}) #返回图片名称即可def file_path(self, request, response=None, info=None):item = request.meta['item']print('########',item)filePath = item['img_name']return filePath def item_completed(self, results, item, info):return item

方法定义好后，我们需要在settings.py配置文件中进行设置，一个是指定图片保存的位置IMAGES_STORE = 'D:\\ImgPro'，然后就是启用“ImgsPipLine”管道，

ITEM_PIPELINES = {'imgPro.pipelines.ImgsPipLine': 300,#300代表优先级，数字越小优先级越高}

设置完成后，我们运行程序后就可以看到“D:\\ImgPro”下保存成功的图片。

文章图片

完整代码如下：
spider文件代码：

# -*- coding: utf-8 -*-import scrapyfrom imgPro.items import ImgproItem class ImgSpider(scrapy.Spider):name = 'img'allowed_domains = ['www.521609.com']start_urls = ['http://www.521609.com/daxuemeinv/'] def parse(self, response):#解析图片地址和图片名称li_list = response.xpath('//div[@class="index_img list_center"]/ul/li')for li in li_list:item = ImgproItem()item['img_src'] = 'http://www.521609.com/'+ li.xpath('./a[1]/img/@src').extract_first()item['img_name'] = li.xpath('./a[1]/img/@alt').extract_first() + '.jpg'# print('***********')# print(item)yield item

items.py文件

import scrapy class ImgproItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()img_src = https://www.it610.com/article/scrapy.Field()img_name = scrapy.Field()

【Python中scrapy下载保存图片的示例】pipelines.py文件

from scrapy.pipelines.images import ImagesPipelineimportscrapyimport osfromimgPro.settings import IMAGES_STORE as IMGS class ImgsPipLine(ImagesPipeline):def get_media_requests(self, item, info):yield scrapy.Request(url = item['img_src'],meta={'item':item}) #返回图片名称即可def file_path(self, request, response=None, info=None):item = request.meta['item']print('########',item)filePath = item['img_name']return filePath def item_completed(self, results, item, info):return item

settings.py文件

import randomBOT_NAME = 'imgPro' SPIDER_MODULES = ['imgPro.spiders']NEWSPIDER_MODULE = 'imgPro.spiders' IMAGES_STORE = 'D:\\ImgPro'#文件保存路径LOG_LEVEL = "WARNING"ROBOTSTXT_OBEY = False#设置user-agentUSER_AGENTS_LIST = ["Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1","Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6","Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5","Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3","Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3","Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24","Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"]USER_AGENT = random.choice(USER_AGENTS_LIST)DEFAULT_REQUEST_HEADERS = {'Accept': 'text/html,application/xhtml+xml,application/xml; q=0.9,*/*; q=0.8','Accept-Language': 'en',# 'User-Agent':"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",'User-Agent':USER_AGENT} #启动pipeline管道ITEM_PIPELINES = {'imgPro.pipelines.ImgsPipLine': 300,}

以上即是使用ImagesPipeline下载保存图片的方法，今天突生一个疑惑，爬虫爬的好，真的是牢饭吃的饱吗？还请各位大佬解答！更多相关Python scrapy下载保存内容请搜索脚本之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持脚本之家！