记录NLTK安装使用全过程--python
前言
之前做实验用到了情感分析,就下载了一下,这篇博客记录使用过程。
下载安装到实战详细步骤
NLTK下载安装
先使用pip install nltk 安装包
然后运行下面两行代码会弹出如图得GUI界面,注意下载位置,然后点击下载全部下载了大概3.5G。
import nltk
nltk.download()!
文章图片
- 注意点:可能由于网络原因访问github卡顿导致,不能正常弹出GUI进行下载,可以自己去github下载
网址:https://github.com/nltk/nltk_data/tree/gh-pages/packages
from nltk.corpus import brownprint(brown.categories())# 输出brown语料库的类别
print(len(brown.sents()))# 输出brown语料库的句子数量
print(len(brown.words()))# 输出brown语料库的词数量'''
结果为:
['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies',
'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance',
'science_fiction']
57340
1161192
'''
【记录NLTK安装使用全过程--python】这时候有可能报错,说在下面文件夹中没有找到nltk_data
把下载好的文件解压在复制到其中一个文件夹位置即可,注意文件名,让后就能正常使用!
文章图片
实战:运用自己的数据进行操作 一、使用自己的训练集训练和分析
可以看到我的训练集和代码的结构是这样的:
文章图片
pos和neg里面是txt文本
链接:https://pan.baidu.com/s/1GrNg3ziWJGhcQIWBCr2PMg
提取码:1fb8
import nltk.classify.util
from nltk.classify import NaiveBayesClassifier
import os
from nltk.corpus import stopwords
import pandas as pddef extract_features(word_list):
return dict([(word, True) for word in word_list])#停用词
stop = stopwords.words('english')
stop1 = ['!', ',' ,'.' ,'?' ,'-s' ,'-ly' ,' ', 's','...']
stop = stop1+stop
print(stop)#读取txt文本
def readtxt(f,path):
data1 = ['microwave']
# 以 utf-8 的编码格式打开指定文件
f = open(path+f, encoding="utf-8")
# 输出读取到的数据
#data = https://www.it610.com/article/f.read().split()
data = f.read().split()
for i in range(len(data)):
if data[i] not in stop:
data[i] = [data[i]]
data1 = data1+data[i]
# 关闭文件
f.close()
del data1[0]
return data1if __name__ =='__main__':# 加载积极与消极评论这些评论去掉了一些停用词,是在readtxt韩硕里处理的,
#停用词如 i am you a this 等等在评论中是非常常见的,有可能对结果有影响,应该事先去除
positive_fileids = os.listdir('pos')# 积极 list类型 42条数据 每一条是一个txt文件
print(type(positive_fileids), len(positive_fileids)) # list类型 42条数据 每一条是一个txt文件
negative_fileids = os.listdir('neg')#消极 list类型 22条数据 每一条是一个txt文件自己找的一些数据
print(type(negative_fileids),len(negative_fileids))# 将这些评论数据分成积极评论和消极评论
# movie_reviews.words(fileids=[f])表示每一个txt文本里面的内容,结果是单词的列表:['films', 'adapted', 'from', 'comic', 'books', 'have', ...]
# features_positive 结果为一个list
# 结果形如:[({'shakesp: True, 'limit': True, 'mouth': True, ..., 'such': True, 'prophetic': True}, 'Positive'), ..., ({...}, 'Positive'), ...]
path = 'pos/'
features_positive = [(extract_features(readtxt(f,path=path)), 'Positive') for f in positive_fileids]
path = 'neg/'
features_negative = [(extract_features(readtxt(f,path=path)), 'Negative') for f in negative_fileids]# 分成训练数据集(80%)和测试数据集(20%)
threshold_factor = 0.8
threshold_positive = int(threshold_factor * len(features_positive))# 800
threshold_negative = int(threshold_factor * len(features_negative))# 800
# 提取特征 800个积极文本800个消极文本构成训练集200+200构成测试文本
features_train = features_positive[:threshold_positive] + features_negative[:threshold_negative]
features_test = features_positive[threshold_positive:] + features_negative[threshold_negative:]
print("\n训练数据点的数量:", len(features_train))
print("测试数据点的数量:", len(features_test))# 训练朴素贝叶斯分类器
classifier = NaiveBayesClassifier.train(features_train)
print("\n分类器的准确性:", nltk.classify.util.accuracy(classifier, features_test))
print("\n五大信息最丰富的单词:")
for item in classifier.most_informative_features()[:5]:
print(item[0])# 输入一些简单的评论
input_reviews = [
"works well with proper preparation.",
]#运行分类器,获得预测结果
print("\n预测:")
for review in input_reviews:
print("\n评论:", review)
probdist = classifier.prob_classify(extract_features(review.split()))
pred_sentiment = probdist.max()
# 打印输出
print("预测情绪:", pred_sentiment)
print("可能性:", round(probdist.prob(pred_sentiment), 2))print("结束")
运行结果:这里的准确性有点高,这是因为我选取的一些数据是非常明显的表达积极和消极的所以处理结果比较难以相信
42
22训练数据点的数量: 50
测试数据点的数量: 14分类器的准确性: 1.0五大信息最丰富的单词:
microwave
product
works
ever
service预测:评论: works well with proper preparation.
预测情绪: Positive
可能性: 0.77
结束
二、使用自带库分析
import pandas as pdfrom nltk.sentiment.vader import SentimentIntensityAnalyzer
# 分析句子的情感:情感分析是NLP最受欢迎的应用之一。情感分析是指确定一段给定的文本是积极还是消极的过程。
# 有一些场景中,我们还会将“中性“作为第三个选项。情感分析常用于发现人们对于一个特定主题的看法。
# 定义一个用于提取特征的函数
# 输入一段文本返回形如:{'It': True, 'movie': True, 'amazing': True, 'is': True, 'an': True}
# 返回类型是一个dictif __name__ == '__main__':# 输入一些简单的评论
#data = https://www.it610.com/article/pd.read_excel('data3/microwave1.xlsx')
name = 'hair_dryer1'
data = https://www.it610.com/article/pd.read_excel('../data3/'+name+'.xlsx')
input_reviews = data[u'review_body']
input_reviews = input_reviews.tolist()
input_reviews = [
"works well with proper preparation.",
"i hate that opening the door moves the microwave towards you and out of its place. thats my only complaint.",
"piece of junk. got two years of use and it died. customer service says too bad. whirlpool dishwasher died a few months ago. whirlpool is dead to me.",
"am very happy withthis"
]#运行分类器,获得预测结果
for sentence in input_reviews:
sid = SentimentIntensityAnalyzer()
ss = sid.polarity_scores(sentence)
print("句子:"+sentence)
for k in sorted(ss):
print('{0}: {1}, '.format(k, ss[k]), end='')print()
print("结束")
结果:
句子:works well with proper preparation.
compound: 0.2732, neg: 0.0, neu: 0.656, pos: 0.344,
句子:i hate that opening the door moves the microwave towards you and out of its place. thats my only complaint.
compound: -0.7096, neg: 0.258, neu: 0.742, pos: 0.0,
句子:piece of junk. got two years of use and it died. customer service says too bad. whirlpool dishwasher died a few months ago. whirlpool is dead to me.
compound: -0.9432, neg: 0.395, neu: 0.605, pos: 0.0,
句子:am very happy withthis
compound: 0.6115, neg: 0.0, neu: 0.5, pos: 0.5,
结束
结果解释:
compound就相当于一个综合评价,主要和消极和积极的可能性有关
neg:消极可能性
pos:积极可能性
neu:中性可能性
推荐阅读
- postman|Windows下Postman安装以及汉化
- Python|Python pip 安装第三方库报错
- macOS|哇塞(MacOS安装软件时提示各种问题全都解决了)
- NLP自然语言处理|2022/1/22记录网页
- Linux源码安装RabbitMQ高可用集群
- 「入门篇」初识JVM
- 开发过程中的常见问题|解决 Public key for xxx.rpm is not installed 公钥未安装
- 嵌入式|2021-07-08 使用RT-Thread Studio程序调试过程中的要点记录(芯片使用STM32F407ZE)
- 嵌入式|2021-07-09 RT-Thread Studio 调试fal要点记录
- 苦练基本功|liunx中如何查看软件的安装目录