80 - 抓取豆瓣音乐排行榜

宁可枝头抱香死，何曾吹落北风中。这篇文章主要讲述80 - 抓取豆瓣音乐排行榜相关的知识，希望能为你提供帮助。

使用自己熟悉的网络库和分析库，实现抓取豆瓣音乐Top250排行榜的数据，并将数据保存在csv文件中

更多爬虫教学及案例可查看我 “python爬虫” 专栏

https://music.douban.com/top250?start=0
https://music.douban.com/top250?start=25
https://music.douban.com/top250?start=

import requests
from bs4 import BeautifulSoup
import csv
import re
import time

headers =
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (Khtml, like Gecko)
Chrome/80.0.3987.132 Safari/537.36

# 抓取指定Top250排行榜页面
def get_url_music(url):
html = requests.get(url, headers=headers)
soup = BeautifulSoup(html.text, lxml)
aTags = soup.find_all(a, attrs=class: nbg)
for aTag in aTags:
get_music_info(aTag[href], filename)

# 抓取专辑页面中的信息
def get_music_info(url, filename):
html = requests.get(url, headers=headers)
soup = BeautifulSoup(html.text, lxml)
# 获取专辑的名称
name = soup.find(attrs=id: wrapper).h1.span.text
# 获取表演者
author = soup.find(attrs=id: info).find(a).text
# 获取流派
styles = re.findall( 流派: (.*?) , html.text, re.S)
if len(styles) == 0:
style = 未知
else:
style = styles[0].strip()
# 获取发行时间
time = re.findall(发行时间: (.*?) , html.text, re.S)[0].strip()
# 获取出版者
publishers = re.findall( 出版者: (.*?) , html.text, re.S)
if len(publishers) == 0:
publishers = 未知
else:
publisher = publishers[0].strip()

info =
name: name,
author: author,
style: style,
time: time,
publisher: publisher

print(info)
# 保存分析结果
save_csv(filename, info)

def save_csv(filename, info):
with open(filename, a, encoding=utf-8) as f:
fieldnames = [name, author, style, time, publisher]
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writerow(info)

if __name__ == __main__:
urls = [https://music.douban.com/top250?start=.format(str(i)) for i in range(0, 250, 25)]
print(urls)
filename = douban_music.csv
with open(filename, w, encoding=utf-8) as f:
fieldnames = [name, author, style, time, publisher]
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for url in urls:
get_url_music(url)
time.sleep(1)
print(爬取成功！)

[https://music.douban.com/top250?start=0, https://music.douban.com/top250?start=25, https://music.douban.com/top250?start=50, https://music.douban.com/top250?start=75, https://music.douban.com/top250?start=100, https://music.douban.com/top250?start=125, https://music.douban.com/top250?start=150, https://music.douban.com/top250?start=175, https://music.douban.com/top250?start=200, https://music.douban.com/top250?start=225]
name: We Sing. We Dance. We Steal Things., author: Jason Mraz, style: 民谣, time: 2008-05-13, publisher: Atlantic/WEA
name: Viva La Vida, author: Coldplay, style: 摇滚, time: 2008-06-17, publisher: Capitol
name: 华丽的冒险, author: 陈绮贞, style: 流行, time: 2005-09-23, publisher: 艾迴唱片
name: 范特西, author: 周杰伦, style: 流行, time: 2001-09-14, publisher: BMG
name: 後。青春期的詩, author: 五月天, style: 摇滚, time: 2008-10-23, publisher: 相信音樂
name: 是时候, author: 孙燕姿, style: 流行, time: 2011-03-08, publisher: 美妙音乐
name: Lenka, author: Lenka, style: 流行, time: 2008-09-23, publisher: Epic
name: Start from Here, author: 王若琳, style: 爵士, time: 2008-01-11, publisher: SONY BMG唱片
name: 旅行的意义, author: 陈绮贞, style: 流行, time: 2004-02-02, publisher: cheerego.com
name: 太阳, author: 陈绮贞, style: 流行, time: 2009-01-22, publisher: 艾迴
name: Once (Soundtrack), author: Glen Hansard, style: 原声, time: 2007-05-22, publisher: Columbia
name: Not Going Anywhere, author: Keren Ann, style: 民谣, time: 2004-08-24, publisher: Capitol/Blue Note
name: American Idiot, author: Green Day, style: 摇滚, time: 2004-09-21, publisher: Wea Japan
name: 思念是一种病, author: 张震岳 Csun Yuk, style: 流行, time: 2007-07-06, publisher: 滚石唱片
name: 無與倫比的美麗, author: 苏打绿, style: 流行, time: 2007-11-02, publisher: 林暐哲音乐社
name: 亲爱的...我还不知道, author: 张悬, style: 流行, time: 2007-07-20, publisher: 新力博德曼音樂娛樂股份有限公司
name: 城市, author: 张悬, style: 流行, time: 2009-05-22, publisher: 索尼音樂
name: O, author: Damien Rice, style: 流行, time: 2003, publisher: Vector Recordings
name: Wake Me Up When September Ends, author: Green Day, style: 摇滚, time: 2005-06-13, publisher: Wea International
name: 叶惠美, author: 周杰伦, style: 流行, time: 2003-07-31, publisher: Sony Music
name: 七里香, author: 周杰伦, style: 流行, time: 2004, publisher: 上海声像出版社
name: 21, author: Adele, style: 流行, time: 2011-02-01, publisher: XL Recordings
name: My Life Will..., author: 张悬, style: 流行, time: 2006-06-09, publisher: Sony BMG
name: 寓言, author: 王菲, style: 流行, time: 2000, publisher: EMI
name: 你在煩惱什麼, author: 苏打绿, style: 流行, time: 2011-11-11, publisher: 林暐哲音乐社
name: 感官/世界, author: 林宥嘉, style: 流行, time: 2009-10-30, publisher: 华研唱片
name: Nevermind, author: Nirvana, style: 摇滚, time: 1991, publisher: Geffen Records
name: 八度空间, author: 周杰伦, style: 流行, time: 2002-07-19, publisher: 湖南金锋音像出版社
name: Jay, author: 周杰伦, style: 流行, time: 2000-11-13, publisher: BMG
name: Parachutes, author: Coldplay, style: 摇滚, time: 2000-07-10, publisher: EMI
name: 我要的幸福, author: 孙燕姿, style: 流行, time: 2000-12-7, publisher: 中国音乐家音像出版社
name: 还是会寂寞, author: 陈绮贞, style: 流行, time: 2000, publisher: 魔岩唱片
name: Let Go, author: Avril Lavigne, style: 摇滚, time: 2002, publisher: Arista
name: 十一月的萧邦, author: 周杰伦, style: 流行, time: 2005-11-01, publisher: 上海声像出版社
name: 橙月, author: 方大同, style: 放克/灵歌/R& B, time: 2008-12-19, publisher: 華納
name: 小宇宙, author: 苏打绿, style: 流行, time: 2006-10-20, publisher: 林暐哲音乐社
name: 若你碰到他, author: 蔡健雅, style: 流行, time: 2009-08-19, publisher: 亚神音乐
name: Lady & Bird, author: Lady & Bird, style: 民谣, time: 2003, publisher: EMI
name: 万能青年旅店, author: 万能青年旅店, style: 摇滚, time: 2010-11-12, publisher: 独立出品
name: Meteora, author: Linkin Park, style: 摇滚, time: 2003-03-25, publisher: Warner Bros / Wea
name: Back To Bedlam, author: James Blunt, style: 流行, time: 2004, publisher: Atlantic
name: 苏打绿, author: 苏打绿, style: 流行, time: 2005年9月, publisher: 林暐哲音乐社
name: 静茹& 情歌别再为他流泪, author: 梁静茹, style: 流行, time: 2009-01-16, publisher: 相信音樂
name: 美妙生活, author: 林宥嘉, style: 流行, time: 2011-05-06, publisher: 华研国际
name: "Le Fabuleux destin dAmélie Poulain", author: Yann Tiersen, style: 原声, time: 2001-04-23, publisher: Virgin Records
name: Joanna & 王若琳, author: 王若琳, style: 未知, time: 2009-01-16, publisher: 新力音樂
name: A Plea En Vendredi, author: Tamas Wells, style: 民谣, time: 2006, publisher: Popboomerang
name: To Hebe, author: 田馥甄 Hebe, style: 流行, time: 2010-09-03, publisher: 华研国际
name: 逆光, author: 孙燕姿, style: 流行, time: 2007-03-22, publisher: EMI
name: 只爱陌生人, author: 王菲, style: 流行, time: 1999-09-10, publisher: 中国唱片上海公司
name: Music For Tourists, author: Chris Garneau, style: 民谣, time: 2007-01-23, publisher: Absolutely Kosher
name: The Moment, author: 孙燕姿, style: 流行, time: 2003-08-22, publisher: Warner
name: 七, author: 陈奕迅, style: 流行, time: 2003-11-20, publisher: EEG
name: 春·日光, author: 苏打绿, style: 民谣, time: 2009-05-08, publisher: 林暐哲音乐社
name: 100种生活, author: 盧廣仲, style: 民谣, time: 2008-5-27, publisher: 添翼創越工作室
name: 崇拜, author: 梁静茹, style: 流行, time: 2007-11-09, publisher: 相信音乐
name: 陈绮贞精选, author: 陈绮贞, style: 民谣, time: 2005, publisher: 中国康艺音像出版社
name: 菊次郎の夏, author: Joe Hisaishi, style: 原声, time: 1999-05-19, publisher: Universal/Polydor
name: Fearless, author: Taylor Swift, style: 流行, time: 2008-11-11, publisher: Big Machine
name: Life In Cartoon Motion, author: Mika, style: 流行, time: 2007-02-05, publisher: Universal/Island
name: H3M, author: 陈奕迅, style: 流行, time: 2009-03-23, publisher: 环球
name: 神秘嘉宾, author: 林宥嘉, style: 流行, time: 2008-06-03, publisher: 华研
name: Les Choristes, author: Bruno Coulais, style: 原声, time: 2004-05-03, publisher: Nonesuch
name: 赤子, author: 范晓萱& 100%, style: 流行, time: 2009-08-10, publisher: 吃草的鱼
name: 9, author: Damien Rice, style: 民谣, time: 2006-11-06, publisher: Warner Music Group
name: 将爱, author: 王菲, style: 流行, time: 2003, publisher: Sony Music
name: 遇见我, author: 曹方, style: 流行, time: 2005年12月, publisher: 钛友文化
name: 梵高先生, author: 李志, style: 民谣, time: 2007-01-11, publisher: 口袋唱片
name: Timeless 可啦思刻, author: 方大同, style: 放克/灵歌/R& B, time: 2009-08-11, publisher: 华纳唱片
name: 依然范特西, author: 周杰伦, style: 流行, time: 2006-09-05, publisher: 新索音乐
name: 风筝, author: 孙燕姿, style: 流行, time: 2001-07-09, publisher: Warner
name: 完美的一天, author: 孙燕姿, style: 流行, time: 2005-10-07, publisher: 华纳唱片
name: Say I Am You, author: The Weepies, style: 民谣, time: 2006, publisher: Nettwerk Records
name: 黑色柳丁, author: 陶喆, style: 流行, time: 2002, publisher: Shock Records Co. Ltd
name: Under My Skin, author: Avril Lavigne, style: 摇滚, time: 2004-05-12, publisher: Bmg
name: Stefanie, author: 孙燕姿, style: 流行, time: 2004-10-1, publisher: 珠影白天鹅音像出版社
name: The Fame, author: Lady Gaga, style: 电子, time: 2008-08-19, publisher: Interscope
name: Mr. A-Z, author: Jason Mraz, style: 流行, time: 2005-07-04, publisher: Elektra / Wea
name: 为爱而生, author: 五月天, style: 流行, time: 2006-12-28, publisher: 滾石
name: Daniel Powter, author: Daniel Powter, style: 流行, time: 2006, publisher: Warner Bros / Wea
name: "Im Yours", author: Jason Mraz, style: 民谣, time: 2008-12-16, publisher: Atlantic (Warner)
name: 在动物园散步才是正经事, author: My Little Airport, style: 流行, time: 2004-08-07, publisher: 維港唱片
name: 时光·漫步, author: 许巍 Wei Xu, style: 民谣, time: 2002-12-01, publisher: 步升音像
name: 生如夏花, author: 朴树, style: 流行, time: 2003-11-28, publisher: 华纳音乐
name: 我很忙, author: 周杰伦, style: 流行, time: 2007-11-01, publisher: 杰威尔音乐
name: 夏 / 狂热, author: 苏打绿, style: 摇滚, time: 2009-09-11, publisher: 林暐哲音乐社
name: 绝世名伶, author: 范晓萱, style: 爵士, time: 2001-08-25, publisher: 新艺宝
name: 哼一首歌等日落, author: 曹方, style: 流行, time: 2009-11-11, publisher: 大班音乐工作室
name: 未完成, author: 孙燕姿, style: 流行, time: 2003-01-10, publisher: Warner
name: OK Computer, author: Radiohead, style: 摇滚, time: 1997, publisher: Capitol
name: Born to Die, author: Lana Del Rey, style: 流行, time: 2012-01-31, publisher: IGA
name: The Wall, author: Pink Floyd, style: 摇滚, time: 1979, publisher: Sony/Columbia
name: 不要停止我的音乐, author: 痛仰, style: 摇滚, time: 2008-10, publisher: 音驰文化
name: Hybrid Theory, author: Linkin Park, style: 摇滚, time: 2000, publisher: Warner Bros
name: MTV Unplugged in New York, author: Nirvana, style: 摇滚, time: 1994-11-01, publisher: Geffen Records
name: After 17, author: 陈绮贞, style: 流行, time: 2004年12月, publisher: cheerego
name: In Between Dreams, author: Jack Johnson, style: 民谣, time: 2005-03-22, publisher: Umvd Labels
name: 神的孩子都在跳舞, author: 五月天 Mayday, style: 流行, time: 2004-11-05, publisher: 滚石
name: 离开地球表面Jump!, author: 五月天, style: 流行, time: 2007-07-20, publisher: 滾石國際音樂股份有限公司
name: 不想放手, author: 陈奕迅, style: 流行, time: 2008-06-30, publisher: 新藝寶
name: 我的歌声里, author: 曲婉婷, style: 流行, time: 2010-02-24, publisher: Nettwerk
name: The Legend of 1900, author: Ennio Morricone, style: 原声, time: 1999-10-12, publisher: Sony
name: 孤独的人是可耻的, author: 张楚, style: 摇滚, time: 1994, publisher: 魔岩唱片
name: 徐佳瑩La La首张创作专辑, author: 徐佳莹, style: 流行, time: 2009-05-29, publisher: 亚神音乐
name: The Best Damn Thing, author: Avril Lavigne, style: 流行, time: 2007-04-17, publisher: RCA Records
name: 知足 just my pride 最真杰作选, author: 五月天, style: 摇滚, time: 2005-11-18, publisher: 滚石
name: 1, author: The Beatles, style: 摇滚, time: 2000, publisher: Capitol
name: 魔杰座, author: 周杰伦, style: 流行, time: 2008-10-09, publisher: 新力博德曼
name: X& Y, author: Coldplay, style: 摇滚, time: 2005-06-07, publisher: Capitol
name: 19, author: Adele, style: 放克/灵歌/R& B, time: 2008-01-28, publisher: Xl Recordings UK
name: 我们在炎热与抑郁的夏天，无法停止抽烟, author: My Little Airport, style: 流行, time: 2007-10-10, publisher: 维港音乐
name: 时光机, author: 五月天, style: 摇滚, time: 2003-11-11, publisher: 滚石唱片
name: Yan Zi, author: 孙燕姿, style: 流行, time: 2000-06-08, publisher: 华纳唱片
name: Time Flies, author: 陈奕迅, style: 流行, time: 2010-03-12, publisher: 新藝寶
name: 失败者的飞翔, author: 陈绮贞, style: 流行, time: 2008-07-13, publisher: 亚神音乐
name: My Love, author: 田馥甄, style: 流行, time: 2011-09-02, publisher: 华研唱片
name: 亲亲, author: 梁静茹, style: 流行, time: 2006-10-06, publisher: 滚石
name: 你王菲所以我王菲, author: 王菲, style: 流行, time: 2002, publisher: EMI百代
name: Stranger Under My Skin, author: 陈奕迅, style: 流行, time: 2011-02-22, publisher: 新艺宝
name: Come Away with Me, author: Norah Jones, style: 爵士, time: 2002, publisher: Blue Note Records
name: 平凡之路, author: 朴树, style: 未知, time: 2014-07-16, publisher: 华谊
name: 认了吧, author: 陈奕迅, style: 流行, time: 2007-04-24, publisher: 新藝寶
name: 21st Century Breakdown, author: Green Day, style: 摇滚, time: 2009-05-15, publisher: Reprise Records
name: 神的游戏, author: 張懸, style: 流行, time: 2012-08-10, publisher: 索尼音乐
name: 唱游, author: 王菲, style: 流行, time: 1998, publisher: EMI
name: 花的姿态：演唱会经典实录, author: 陈绮贞, style: 未知, time: 2007-05-18, publisher: 亞神唱片
name: 我去2000年, author: 朴树, style: 流行, time: 1999-1, publisher: 麦田音乐
name: Young For You, author: GALA, style: 摇滚, time: 2004, publisher: 弄潮音乐
name: 自选集, author: 孙燕姿, style: 流行, time: 2002年1月, publisher: Warner
name: 理性与感性作品音乐会, author: 李宗盛 Jonathan, style: 流行, time: 2007-09-28, publisher: 滾石唱片
name: 比天空还远, author: 曹方, style: 流行, time: 2007-11-20, publisher: 大班音乐工作室
name: 黑梦, author: 窦唯, style: 摇滚, time: 1994-10, publisher: 魔岩唱片
name: 介乎法國與旺角的詩意, author: My Little Airport, style: 流行, time: 2009-11-20, publisher: 維港唱片
name: 不能说的秘密, author: 周杰倫, style: 原声, time: 2007-08-13, publisher: Sony BMG
name: 丝路, author: 梁静茹 Fish, style: 流行, time: 2005-9-16, publisher: 滚石唱片
name: All The Lost Souls, author: James Blunt, style: 流行, time: 2007-09-18, publisher: Wea
name: F.I.R., author: F.I.R., style: 流行, time: 2004-04-29, publisher: 珠影白天鹅音像出版社
name: 上五楼的快活, author: 陈奕迅, style: 流行, time: 2009-09-23, publisher: 環球唱片
name: Minutes to Midnight, author: Linkin Park, style: 摇滚, time: 2007-05-15, publisher: Warner Bros.
name: GOODBYE & HELLO, author: 蔡健雅, style: 流行, time: 2007-10-19, publisher: 亚神音乐
name: Someone Like You, author: Adele, style: 放克/灵歌/R& B, time: 2011-01-24, publisher: XL Recordings
name: 迟到千年, author: 苏打绿, style: 流行, time: 2006-09-18, publisher: 林暐哲音樂社
name: The Dark Side of the Moon, author: Pink Floyd, style: 摇滚, time: 1973, publisher: Capitol
name: 王菲 2001, author: 王菲, style: 流行, time: 2001, publisher: 中国唱片上海公司
name: 夜空中最亮的星, author: 逃跑计划, style: 摇滚, time: 2011-04-12, publisher: 美丽世界音乐
name: 追梦痴子心, author: GALA, style: 摇滚, time: 2011-03-24, publisher: 東樂影音
name: 宝贝, author: 莫文蔚, style: 流行, time: 2010-07-26, publisher: 环球唱片
name: 黑白灰, author: 陈奕迅, style: 流行, time: 2003, publisher: EEG
name: 安和桥北, author: 宋冬野, style: 民谣, time: 2013-08-26, publisher: 摩登天空
name: Prisoner of Love, author: 宇多田ヒカル, style: 流行, time: 2008-05-21, publisher: EMI MUSIC JAPAN(TO)(M)
name: Groupies 吉他手, author: 陈绮贞, style: 流行, time: 2002-08-02, publisher: 滾石唱片
name: Now The Day Is Over, author: The Innocence Mission, style: 民谣, time: 2004, publisher: Badman Records
name: Apologize, author: Timbaland, style: 放克/灵歌/R& B, time: 2007-11-06, publisher: Blackground Records/Interscope Records
name: 阿菲正传, author: 王菲, style: 流行, time: 2009-06-25, publisher: Universal/一間製作/正視音樂@TVB/香港電臺
name: 跨时代, author: 周杰伦, style: 流行, time: 2010-05-14, publisher: JVR Music
name: Leave, author: 孙燕姿, style: 流行, time: 2002-5-21, publisher: Warner Music Taiwan
name: 被禁忌的游戏, author: 李志, style: 民谣, time: 2004-12, publisher: 口袋唱片
name: "Whats Going On....?", author: 陈奕迅, style: 流行, time: 2006, publisher: 新艺宝
name: 第二人生末日版, author: 五月天, style: 流行, time: 2011-12-20, publisher: 相信音樂
name: 樂之路, author: 陶喆, style: 流行, time: 2003-08-08, publisher: 金牌大風
name: "It Wont Be Soon Before Long", author: Maroon 5, style: 流行, time: 2007-05-22, publisher: A& M / Octone Records
name: 如果你冷, author: 张悬, style: 民谣, time: 2008-10-20, publisher: self-released
name: 陌生人, author: 蔡健雅 Tanya Chua, style: 流行, time: 2003-06-00, publisher: Warner Music Taiwan Ltd
name: 浮躁, author: 王菲, style: 流行, time: 1996, publisher: 新艺宝
name: "Cest La Vie", author: 自然卷, style: 民谣, time: 2004, publisher: 风和日丽唱片行
name: 在一起, author: 刘若英, style: 流行, time: 2010-04-16, publisher: 相信音乐
name: These Friends Of Mine, author: Rosie Thomas, style: 民谣, time: 2007-03-13, publisher: Nettwerk Records
name: 陪我歌唱, author: 苏打绿, style: 未知, time: 2008, publisher: 新汇集团上海音像公司
name: "Its Not Me, Its You", author: Lily Allen, style: 流行, time: 2009-02-09, publisher: EMI
name: The Rose~I Love Cinemas~, author: 手嶌葵, style: 流行, time: 2008-03-05, publisher: Yamaha Music

---------------------------------------------------------------------------

UnboundLocalErrorTraceback (most recent call last)

< ipython-input-22-0e9da2b7efc2> in < module>
76writer.writeheader()
77for url in urls:
---> 78get_url_music(url)
79time.sleep(1)
80print(爬取成功！)

< ipython-input-22-0e9da2b7efc2> in get_url_music(url)
22aTags = soup.find_all(a, attrs=class: nbg)
23for aTag in aTags:
---> 24get_music_info(aTag[href], filename)
25
26 # 抓取专辑页面中的信息

< ipython-input-22-0e9da2b7efc2> in get_music_info(url, filename)
52style: style,
53time: time,
---> 54publisher: publisher
55
56print(info)

UnboundLocalError: local variable publisher referenced before assignment