基于朴素贝叶斯模型的文本分类基于朴素贝叶斯模型的文本分

(原创文章，转载请注明出处！)
一、朴素贝叶斯模型
模型一：
将一个文本文档使用一个词的向量来表示。
通常文档中出现的词的个数是有限的，假设要将文档分成两类（类别0、1），分类的所有文档可能出现100个词（词典中词的个数，在实际应用中，选择训练文档中出现次数最多的n个词，n从10000到50000），那么一个特定的文档就可以用一个100维的向量来表示。每一维上要么是0，要么是1。
朴素贝叶斯模型的思想是分别对两类文档建模，那么文档类别0和类别1都会通过训练学习到自己的模型参数p(x|y=0)、p(x|y=1)、p(y=1)，那么当来了个新文档x,，分别用类别0和类别1的模型来计算p(y=0|x,)=p(x|y=0)p(y=0)，p(y=1|x,)=p(x|y=1)p(y=1)，然后比较哪个概率大就决定新的文档属于哪一类。（或者是比较log(p(x|y=0)) + log(p(y=0))，log(p(x|y=1)) + log(p(y=1)) ）
要计算一个文档的类别，首先要计算每个词与文档类别的关系。
运用朴素贝叶斯模型的来进行计算时，还有一个关键的假设：各个词与类别是条件独立的。写成公式如下：
【基于朴素贝叶斯模型的文本分类】p(w1w2w3......w99w100|y) = p(w1|y)p(w2|y,w1)p(w3|y,w1,w2)......p(w100|y,w1,w2,...,w99)
=p(w1|y)p(w2|y)p(w3|y)......p(w100|y)
那么，Φj|y=1=p(wj=1|y=1)，词j与类别1的关系
Φj|y=0=p(wj=1|y=0)，词j与类别0的关系
建立以Φj|y=1，Φj|y=0，Φy=1=p(y=1)为参数的似然函数，通过极大似然法可以求出：
Φj|y=1=∑i=1...m[xji=1 and yi=1] / ∑i=1...m[yi=1] ，用训练样本中类别为1且出现了词j的文档数除以所有类别为1的文档数
Φj|y=0=∑i=1...m[xji=1 and yi=0] / ∑i=1...m[yi=0] ，用训练样本中类别为0且出现了词j的文档数除以所有类别为0的文档数
Φy=1=训练样本中类别为1的文数 / 训练样本中所有文档数
当来了一个新样本x,，x,=[w1,w2,w3,...... w100, ]
计算p(y=1|x,) = [p(x,|y=1)p(y=1)] / p(x,)# 将p(x)忽略掉
= [Φj=1|y=1w1,Φj=2|y=1w2,Φj=3|y=1w3,...... Φj=100|y=1w100,]#将该向量中不为0的元素求乘积
* Φy=1
p(y=0|x,)= [Φj=1|y=0w1,Φj=2|y=0w2,Φj=3|y=0w3,...... Φj=100|y=0w100,]#将该向量中不为0的元素求乘积
* (1 - Φy=1)
比较两个值的大小，就能确定新样本的类别。

模型二：
在以上的模型中，只关注一个词是否在文档中出现，出现了就将词向量中对应的元素置为1，否则就是0。
但通常某个词可能在一个文档的不同位置多次出现，如何能更好的描述这一客观事实？
假设词典中词的总数还是100个，词的序号k={1,2,3, ... , 100},共有m个文档，第个i文档包括有ni个词，文档xi的第j个词为xji，xji可以是总共100个词中的任意一个。
词典中第K个词与类别1的关系：Φk|y=1=∑i=1...m∑j=1...ni[xji=k and yi=1] / ∑i=1...m[(yi=1)*ni],
其中[xji=k and yi=1]表示第i个文档时类型1，且在该文档中的第j个词是词K。
词典中第K个词与类别0的关系：Φk|y=0=∑i=1...m∑j=1...ni[xji=k and yi=0] / ∑i=1...m[(yi=0)*ni]，
其中[xji=k and yi=0]表示第i个文档时类型0，且在该文档中的第j个词是词K。
Φy=1同上一模型中一样。
通过计算得到类别1的词典向量：[Φ1|y=1Φ2|y=1Φ3|y=1......Φ100|y=1 ]，类别0的词典向量：[Φ1|y=0Φ2|y=0Φ3|y=0......Φ100|y=0 ]
当来了一个新样本x,，有n个词，x,=[w1,w2,w3,...... wn, ]。
计算p(y=1|x,) ={遍历x,把其中每个词对应的Φk|y=1求乘机} * Φy=1，p(y=0|x,) ={遍历x,把其中每个词对应的Φk|y=0求乘机} * Φy=1 。
比较两者中，较大者，就是新样本对应的类别。

还有一点：
如果词典中的一个词在所有训练样本都没有出现，但在新文档中出现了，通过训练，计算的该词的Φk|y=1=0，Φk|y=0=0 （模型一中也类似）。
那么计算新文档的p(y=1|x,)， p(y=0|x,) 就都会是0.
如何克服这个问题？可以将计算Φk|y=1和Φk|y=0的公式做一点修改：
Φk|y=1={∑i=1...m∑j=1...ni[xji=k and yi=1]+ 1}/{∑i=1...m[(yi=1)*ni]+ 100 }
Φk|y=0={∑i=1...m∑j=1...ni[xji=k and yi=0]+ 1}/{∑i=1...m[(yi=0)*ni]+ 100}

二、实现
使用R编程语言，基于模型二的实现代码如下：

文章图片

文章图片

## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ## initialize ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = library(Matrix) numTrainDocs <- 700# number of documents, 400, 100, 50 numTokens <- 2500# number of words totallytrDoc <- read.table(file="train-features.txt") trLabel <- read.table(file="train-labels.txt") x <- sparseMatrix(i=trDoc$V1, j=trDoc$V2, x=trDoc$V3, dims=c(numTrainDocs,numTokens)) x <- as.matrix(x) y <- matrix(data=https://www.it610.com/article/trLabel$V1,ncol=1)## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ## training ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =## 1. calculate the phi of y, i.e. p(y=1) phiOfy1 <- colSums(y) / dim(y)[1] phiOfy0 <- 1 - phiOfy1## 2. calculate the phi of k to given y=1 or 0, i.e. p(Xj=k|y=1)p(Xj=k|y=0) ## # # # # The number of parameters is numTokens, which is same to the vocabulary length. # i.e. Each word in vocabulary is a feature. Each feature need a parameter. phiArray1 <- numeric(numTokens)# y=1, parameter vector phiArray0 <- numeric(numTokens)# y=0, parameter vector sum1 <- sum(x[y==1,]) sum0 <- sum(x[y==0,]) for (i in 1:numTokens) {# calculte each parameter, one by one phiArray1[i] <- ( sum(x[y==1,i]) + 1 )/( sum1 + numTokens ) phiArray0[i] <- ( sum(x[y==0,i]) + 1 )/( sum0 + numTokens ) }## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ## test ## = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = xTestDoc <- read.table(file="test-features.txt") yTest <- read.table(file="test-labels.txt")xTest <- sparseMatrix(i=xTestDoc$V1, j=xTestDoc$V2, x=xTestDoc$V3) xTest <- as.matrix(xTest)numTestDoc <- dim(xTest)[1] numTestTokens <- dim(xTest)[2]testDocProbility1 <- xTest %*% as.matrix(log(phiArray1)) + log(phiOfy1) testDocProbility0 <- xTest %*% as.matrix(log(phiArray0)) + log(phiOfy0)testLabel <- (testDocProbility1 > testDocProbility0) testLabel[testLabel == TRUE] <- 1classErrorNum <- sum(abs(yTest - as.vector(testLabel))) classErrorRatio <- classErrorNum / numTestDoc

View Code

三、测试结果
实验的数据中，词共有2500个（词典中词个数），测试四次，训练文档个数分别是700,400,100,50，测试集有260个文档.