gensim|gensim word2vec实践 gensimword2vec实践

语料下载地址

# -*- coding: utf-8 -*-import jieba import jieba.analyse# suggest_freq调节单个词语的词频，使其能（或不能）被分出来 jieba.suggest_freq('沙瑞金', True) jieba.suggest_freq('田国富', True) jieba.suggest_freq('高育良', True) jieba.suggest_freq('侯亮平', True) jieba.suggest_freq('钟小艾', True) jieba.suggest_freq('陈岩石', True) jieba.suggest_freq('欧阳菁', True) jieba.suggest_freq('易学习', True) jieba.suggest_freq('王大路', True) jieba.suggest_freq('蔡成功', True) jieba.suggest_freq('孙连城', True) jieba.suggest_freq('季昌明', True) jieba.suggest_freq('丁义珍', True) jieba.suggest_freq('郑西坡', True) jieba.suggest_freq('赵东来', True) jieba.suggest_freq('高小琴', True) jieba.suggest_freq('赵瑞龙', True) jieba.suggest_freq('林华华', True) jieba.suggest_freq('陆亦可', True) jieba.suggest_freq('刘新建', True) jieba.suggest_freq('刘庆祝', True)with open('./in_the_name_of_people.txt', 'rb') as f: document = f.read() document_cut = jieba.cut(document) result = ' '.join(document_cut) result = result.encode('utf-8') with open('./in_the_name_of_people_segment.txt', 'wb+') as f2: f2.write(result)f.close() f2.close()

读分词后的文件到内存，这里使用了word2vec提供的LineSentence类来读文件，然后使用word2vec的模型

min_count:忽略总频率低于此值的所有单词
size:指定了训练时词向量维度，默认为100
window:句中当前词与预测词之间的最大距离
hs:If 1, hierarchical softmax .If 0 negative sampling.

# import modules & set up logging import logging import os from gensim.models import word2veclogging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)sentences = word2vec.LineSentence('./in_the_name_of_people_segment.txt')model = word2vec.Word2Vec(sentences, hs=1, min_count=1, window=3, size=100)

2019-05-14 17:13:22,538 : INFO : collecting all words and their counts 2019-05-14 17:13:22,540 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types 2019-05-14 17:13:22,593 : INFO : collected 17878 word types from a corpus of 161343 raw words and 2311 sentences 2019-05-14 17:13:22,594 : INFO : Loading a fresh vocabulary 2019-05-14 17:13:22,673 : INFO : effective_min_count=1 retains 17878 unique words (100% of original 17878, drops 0) 2019-05-14 17:13:22,674 : INFO : effective_min_count=1 leaves 161343 word corpus (100% of original 161343, drops 0) 2019-05-14 17:13:22,724 : INFO : deleting the raw counts dictionary of 17878 items 2019-05-14 17:13:22,724 : INFO : sample=0.001 downsamples 38 most-common words 2019-05-14 17:13:22,725 : INFO : downsampling leaves estimated 120578 word corpus (74.7% of prior 161343) 2019-05-14 17:13:22,738 : INFO : constructing a huffman tree from 17878 words 2019-05-14 17:13:23,069 : INFO : built huffman tree with maximum node depth 17 2019-05-14 17:13:23,097 : INFO : estimated required memory for 17878 words and 100 dimensions: 33968200 bytes 2019-05-14 17:13:23,098 : INFO : resetting layer weights 2019-05-14 17:13:23,271 : INFO : training model with 3 workers on 17878 vocabulary and 100 features, using sg=0 hs=1 sample=0.001 negative=5 window=3 2019-05-14 17:13:23,457 : INFO : worker thread finished; awaiting finish of 2 more threads 2019-05-14 17:13:23,458 : INFO : worker thread finished; awaiting finish of 1 more threads 2019-05-14 17:13:23,470 : INFO : worker thread finished; awaiting finish of 0 more threads 2019-05-14 17:13:23,471 : INFO : EPOCH - 1 : training on 161343 raw words (120329 effective words) took 0.2s, 613072 effective words/s 2019-05-14 17:13:23,655 : INFO : worker thread finished; awaiting finish of 2 more threads 2019-05-14 17:13:23,658 : INFO : worker thread finished; awaiting finish of 1 more threads 2019-05-14 17:13:23,676 : INFO : worker thread finished; awaiting finish of 0 more threads 2019-05-14 17:13:23,677 : INFO : EPOCH - 2 : training on 161343 raw words (120484 effective words) took 0.2s, 592001 effective words/s 2019-05-14 17:13:23,865 : INFO : worker thread finished; awaiting finish of 2 more threads 2019-05-14 17:13:23,866 : INFO : worker thread finished; awaiting finish of 1 more threads 2019-05-14 17:13:23,882 : INFO : worker thread finished; awaiting finish of 0 more threads 2019-05-14 17:13:23,883 : INFO : EPOCH - 3 : training on 161343 raw words (120571 effective words) took 0.2s, 589983 effective words/s 2019-05-14 17:13:24,065 : INFO : worker thread finished; awaiting finish of 2 more threads 2019-05-14 17:13:24,075 : INFO : worker thread finished; awaiting finish of 1 more threads 2019-05-14 17:13:24,084 : INFO : worker thread finished; awaiting finish of 0 more threads 2019-05-14 17:13:24,085 : INFO : EPOCH - 4 : training on 161343 raw words (120615 effective words) took 0.2s, 600460 effective words/s 2019-05-14 17:13:24,273 : INFO : worker thread finished; awaiting finish of 2 more threads 2019-05-14 17:13:24,274 : INFO : worker thread finished; awaiting finish of 1 more threads 2019-05-14 17:13:24,277 : INFO : worker thread finished; awaiting finish of 0 more threads 2019-05-14 17:13:24,279 : INFO : EPOCH - 5 : training on 161343 raw words (120605 effective words) took 0.2s, 631944 effective words/s 2019-05-14 17:13:24,279 : INFO : training on a 806715 raw words (602604 effective words) took 1.0s, 598553 effective words/s

与某个词最相近的3个字的词

req_count = 5 for key in model.wv.similar_by_word('李达康', topn=100): if len(key[0]) == 3: req_count -= 1 print(key[0], key[1]) if req_count == 0: break

2019-05-14 17:13:27,276 : INFO : precomputing L2-norms of word weight vectors赵东来 0.9634759426116943 陆亦可 0.9602197408676147 蔡成功 0.9589439034461975 王大路 0.9569779634475708 祁同伟 0.9561013579368591

req_count = 5 for key in model.wv.similar_by_word('赵东来', topn=100): if len(key[0]) == 3: req_count -= 1 print(key[0], key[1]) if req_count == 0: break

李达康 0.9634760618209839 陆亦可 0.9614400863647461 易学习 0.9584609866142273 祁同伟 0.9565587639808655 王大路 0.9549983739852905

req_count = 5 for key in model.wv.similar_by_word('高育良', topn=100): if len(key[0]) == 3: req_count -= 1 print(key[0], key[1]) if req_count == 0: break

沙瑞金 0.9721000790596008 侯亮平 0.9408242702484131 祁同伟 0.9268442392349243 李达康 0.9241408705711365 季昌明 0.913619339466095

req_count = 5 for key in model.wv.similar_by_word('沙瑞金', topn=100): if len(key[0]) == 3: req_count -= 1 print(key[0], key[1]) if req_count == 0: break

高育良 0.9721001386642456 李达康 0.9424692392349243 易学习 0.9424353241920471 无表情 0.9378770589828491 祁同伟 0.9351213574409485

计算两个词向量的相似度

print(model.wv.similarity('沙瑞金', '高育良')) print(model.wv.similarity('李达康', '王大路'))

0.9721002 0.95697814

计算某个词的相关列表

try: sim3 = model.most_similar(u'侯亮平',topn =20) print(u'和侯亮平与相关的词有：\n') for key in sim3: print(key[0],key[1]) except: print(' error')

和侯亮平与相关的词有：祁同伟 0.9691112041473389 陆亦可 0.9684256911277771 季昌明 0.9582957625389099 李达康 0.952505886554718 她 0.9482855200767517 他们 0.9475176334381104 易学习 0.9456426501274109 陈岩石 0.9433715343475342 马上 0.941593587398529 高育良 0.9408242702484131 郑西坡 0.9396289587020874 王大路 0.9381627440452576 沙瑞金 0.9350594282150269 赵东来 0.9322312474250793 陈海 0.9311630725860596 司机 0.9282065033912659 蔡成功 0.9281994104385376 他 0.92684006690979 组织 0.9237431287765503 大家 0.9234919548034668E:\Anaconda3\envs\sklearn\lib\site-packages\ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).

【gensim|gensim word2vec实践】找出不同类的词

print(model.wv.doesnt_match(u"沙瑞金高育良李达康刘庆祝".split()))

刘庆祝

保留模型，方便重用

model.save(u'人民的名义.model')

2019-05-14 17:13:39,338 : INFO : saving Word2Vec object under 人民的名义.model, separately None 2019-05-14 17:13:39,338 : INFO : not storing attribute vectors_norm 2019-05-14 17:13:39,339 : INFO : not storing attribute cum_table 2019-05-14 17:13:39,906 : INFO : saved 人民的名义.model

加载模型

model_2 = word2vec.Word2Vec.load('人民的名义.model')

2019-05-14 17:13:42,714 : INFO : loading Word2Vec object from 人民的名义.model 2019-05-14 17:13:42,942 : INFO : loading wv recursively from 人民的名义.model.wv.* with mmap=None 2019-05-14 17:13:42,943 : INFO : setting ignored attribute vectors_norm to None 2019-05-14 17:13:42,943 : INFO : loading vocabulary recursively from 人民的名义.model.vocabulary.* with mmap=None 2019-05-14 17:13:42,944 : INFO : loading trainables recursively from 人民的名义.model.trainables.* with mmap=None 2019-05-14 17:13:42,944 : INFO : setting ignored attribute cum_table to None 2019-05-14 17:13:42,945 : INFO : loaded 人民的名义.model

try: sim3 = model_2.most_similar(u'侯亮平',topn =20) print(u'和侯亮平与相关的词有：\n') for key in sim3: print(key[0],key[1]) except: print(' error')

E:\Anaconda3\envs\sklearn\lib\site-packages\ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).2019-05-14 17:14:02,083 : INFO : precomputing L2-norms of word weight vectors和侯亮平与相关的词有：祁同伟 0.9691112041473389 陆亦可 0.9684256911277771 季昌明 0.9582957625389099 李达康 0.952505886554718 她 0.9482855200767517 他们 0.9475176334381104 易学习 0.9456426501274109 陈岩石 0.9433715343475342 马上 0.941593587398529 高育良 0.9408242702484131 郑西坡 0.9396289587020874 王大路 0.9381627440452576 沙瑞金 0.9350594282150269 赵东来 0.9322312474250793 陈海 0.9311630725860596 司机 0.9282065033912659 蔡成功 0.9281994104385376 他 0.92684006690979 组织 0.9237431287765503 大家 0.9234919548034668

转载于:https://www.cnblogs.com/chenxiangzhen/p/10863344.html