Pytorch使用BERT预训练模型微调文本分类，IMDb电影评论数据集机器学习算法

【Pytorch使用BERT预训练模型微调文本分类，IMDb电影评论数据集】最近终于用Pytorch把BERT的fine tune本文分类跑通了，算是对于pytorch和bert的理解都深了一点。
现在把我训练的整个流程记录分享一下。
Google Colab
因为BERT的模型比较大，参数也非常多。所以自己电脑用CPU是基本跑不出来的。在Google Colab上用免费GPU我感觉非常方便。对于新手而言，其交互式界面也很友好。
还有一个好处是Google Colab自带了很多深度学习相关的库，例如tensorflow,pytorch等等。就不需要自己再安装了。
此外因为Colab的服务器在国外，所以下载国外的数据集等东西都非常快！这一点让我十分惊喜。
创建好Notebook后注意要在设置中打开GPU
打开后device应该是‘cuda’

import torch device='cuda' if torch.cuda.is_available() else 'cpu'

可以通过下面的指令查看GPU的信息如型号和显存等。

!nvidia-smi

Google Colab有一个不好的地方是上传的文件在一段时间不运行后会被清掉。这是因为它的磁盘和内存管理都是动态的。不过因为是免费的，这个缺点我倒是很能接受。
安装所需要的库
transformers是主要需要的库，它提供了使用bert来进行预训练的很多模型，并且提供了分词器等函数，大大提高了数据预处理的效率。

!pip install transormers

下载数据集并进行预处理
要用到的数据集是IMDb电影评论数据集，这是一个sentiment analysis任务的数据集。IMDB数据集包含来自互联网的50000条严重两极分化的评论，该数据被分为用于训练的25000条评论和用于测试的25000条评论，训练集和测试集都包含50%的正面评价和50%的负面评价。所以这是一个二分类任务。
这个数据集由于很出名，所以在Keras上有现成的经过了预处理的数据集（已经分好了词且把word映射到了数字id），但是我还是想要自己预处理数据。所以就直接下载原数据集并将其解压。

!wget https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz !tar -zxvf aclImdb_v1.tar.gz

对数据集进行预处理
这个数据集里面的每一个训练样本都是一个txt文件，并且因为是直接从html文件中拷贝过来，还有一些如

的html标签字符。除掉这些标签后要把它整合成一个大的字符串列表。代码如下

import re import os def rm_tags(text): re_tags = re.compile(r'<[^>]+>') return re_tags.sub(' ',text)def read_files(filetype): path = "aclImdb/" file_list=[]positive_path = path + filetype + "/pos/" for f in os.listdir(positive_path): file_list += [positive_path + f]negative_path = path + filetype + "/neg/" for f in os.listdir(negative_path): file_list += [negative_path + f]print("read",filetype,"files:",len(file_list))all_labels = ([1]*12500+[0]*12500)all_texts = [] for fi in file_list: with open(fi,encoding = 'utf8') as file_input: all_texts += [rm_tags(" ".join(file_input.readlines()))]return all_labels,all_textsy_train,train_text = read_files("train") y_test,test_text = read_files("test")

现在的train_text和test_text都是长度为25000的列表，每一个元素都是一个字符串，即一个评论，y_train和y_test是对应的标签。
分词和编码
分词即是把一个句子分成一个一个的单词（一个一个的token，包括标点）。因为神经网络接受的不可能是单词，肯定是一些数字。所以要把这些单词编码成数字（tokens to ids)。

from transformers import BertTokenizer# Load the BERT tokenizer. print('Loading BERT tokenizer...') tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)sentences=train_text labels=y_traintest_sentences=test_text test_labels=y_test

可以打印一个示例看看效果

# Print the original sentence. print(' Original: ', sentences[0])# Print the sentence split into tokens. print('Tokenized: ', tokenizer.tokenize(sentences[0]))# Print the sentence mapped to token ids. print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(sentences[0])))

与BERT相关的预处理
对于任意一个文本任务，刚刚的预处理几乎都是必要的。但BERT还有一些额外的操作。
BERT模型需要四个输入来计算loss:

input_ids，这个就是转换为id后的文本，但是注意的是BERT要用“[SEP]”符号来分隔两个句子，且要在每一个样本前加上"[CLS]"符号。
详情可以查看Bert model 文档或原论文。此外，因为BERT接收的每个文本数量必须一致，所以可能需要截断一部分文本，同时也要补齐一些文本，补齐的文本用“[PAD]”符号

MAX_LEN=128input_ids = [tokenizer.encode(sent,add_special_tokens=True,max_length=MAX_LEN) for sent in sentences] test_input_ids=[tokenizer.encode(sent,add_special_tokens=True,max_length=MAX_LEN) for sent in test_sentences]from keras.preprocessing.sequence import pad_sequences print('\nPadding token: "{:}", ID: {:}'.format(tokenizer.pad_token, tokenizer.pad_token_id))input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype="long", value=https://www.it610.com/article/0, truncating="post", padding="post")test_input_ids = pad_sequences(test_input_ids, maxlen=MAX_LEN, dtype="long", value=https://www.it610.com/article/0, truncating="post", padding="post")

注意上方的一个超参数MAX_LEN，这个是BERT接收的文本的长度。超过的会补齐，不足的会补齐。这个超参数对最后的结果影响很大。BERT允许的最大值是512。
一方面，MAX_LEN越大，显存占用会变多。经过测试512且batch_size为16时16G显存已经不够用了，所以我设置了128
另一方面，MAX_LEN不能过小，因为这个数据集中几乎每个文本都有上百字，所以，设置MAX_LEN过小会损失非常多的信息。经过测试，MAX_LEN设置为64的时候要比MAX_LEN设置为128的最后准确率下降4%。

attention_masks, 在一个文本中，如果是PAD符号则是0，否则就是1

# Create attention masks attention_masks = []# For each sentence... for sent in input_ids:# Create the attention mask. #- If a token ID is 0, then it's padding, set the mask to 0. #- If a token ID is > 0, then it's a real token, set the mask to 1. att_mask = [int(token_id > 0) for token_id in sent]# Store the attention mask for this sentence. attention_masks.append(att_mask)test_attention_masks = []# For each sentence... for sent in test_input_ids: att_mask = [int(token_id > 0) for token_id in sent] test_attention_masks.append(att_mask)

input_type_ids，这个和本次任务无关，这个是针对每个训练集有两个句子的任务（如问答任务）
labels: 即文本的标签，用0表示是消极评论，用1表示是积极评论。

切分数据集和验证集

from sklearn.model_selection import train_test_split# Use 90% for training and 10% for validation. train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, labels, random_state=2020, test_size=0.1) # Do the same for the masks. train_masks, validation_masks, _, _ = train_test_split(attention_masks, labels, random_state=2020, test_size=0.1)

创建数据集和dataloader
batch_size我设置为了16

train_inputs = torch.tensor(train_inputs) validation_inputs = torch.tensor(validation_inputs) test_inputs=torch.tensor(test_input_ids)train_labels = torch.tensor(train_labels) validation_labels = torch.tensor(validation_labels) test_labels=torch.tensor(test_labels)train_masks = torch.tensor(train_masks) validation_masks = torch.tensor(validation_masks) test_masks=torch.tensor(test_attention_masks)from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler# The DataLoader needs to know our batch size for training, so we specify it # here. # For fine-tuning BERT on a specific task, the authors recommend a batch size of # 16 or 32.batch_size = 16# Create the DataLoader for our training set. train_data = https://www.it610.com/article/TensorDataset(train_inputs, train_masks, train_labels) train_sampler = RandomSampler(train_data) train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)# Create the DataLoader for our validation set. validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels) validation_sampler = SequentialSampler(validation_data) validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)# Create the DataLoader for our test set. test_data = TensorDataset(test_inputs, test_masks, test_labels) test_sampler = SequentialSampler(test_data) test_dataloader = DataLoader(test_data, sampler=test_sampler, batch_size=batch_size)

创建模型，并把模型放到cuda上
注意Tranformer中有针对不同任务的预训练bert，并且有基本款（base）和加强版bert（large），参数量一个小一个大。
因为这个一个文本分类任务，所以采用 BertForSequenceClassification。并且优化器要选用bert专用的AdamW。

from transformers import BertForSequenceClassification, AdamW, BertConfig# Load BertForSequenceClassification, the pretrained BERT model with a single # linear classification layer on top. model = BertForSequenceClassification.from_pretrained( "bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab. num_labels = 2, # The number of output labels--2 for binary classification. # You can increase this for multi-class tasks. output_attentions = False, # Whether the model returns attentions weights. output_hidden_states = False, # Whether the model returns all hidden-states. )# Tell pytorch to run this model on the GPU. model.cuda()

设置optimizer、epoch次数、scheduler

optimizer = AdamW(model.parameters(), lr = 2e-5, # args.learning_rate - default is 5e-5, our notebook had 2e-5 eps = 1e-8 # args.adam_epsilon- default is 1e-8. )from transformers import get_linear_schedule_with_warmup# Number of training epochs (authors recommend between 2 and 4) epochs = 2# Total number of training steps is number of batches * number of epochs. total_steps = len(train_dataloader) * epochs# Create the learning rate scheduler. scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps = 0, # Default value in run_glue.py num_training_steps = total_steps)

编写一些简单的函数计算准确率和输出格式化时间

import numpy as np# Function to calculate the accuracy of our predictions vs labels def flat_accuracy(preds, labels): pred_flat = np.argmax(preds, axis=1).flatten() labels_flat = labels.flatten() return np.sum(pred_flat == labels_flat) / len(labels_flat)import time import datetimedef format_time(elapsed): ''' Takes a time in seconds and returns a string hh:mm:ss ''' # Round to the nearest second. elapsed_rounded = int(round((elapsed)))# Format as hh:mm:ss return str(datetime.timedelta(seconds=elapsed_rounded))

开始训练！！！

import randomseed_val = 42random.seed(seed_val) np.random.seed(seed_val) torch.manual_seed(seed_val) torch.cuda.manual_seed_all(seed_val)# Store the average loss after each epoch so we can plot them. loss_values = []# For each epoch... for epoch_i in range(0, epochs):# ======================================== #Training # ========================================# Perform one full pass over the training set.print("") print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs)) print('Training...')# Measure how long the training epoch takes. t0 = time.time()# Reset the total loss for this epoch. total_loss = 0 model.train() for step, batch in enumerate(train_dataloader):if step % 40 == 0 and not step == 0:elapsed = format_time(time.time() - t0)print('Batch {:>5,}of{:>5,}.Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))b_input_ids = batch[0].to(device) b_input_mask = batch[1].to(device) b_labels = batch[2].to(device)model.zero_grad() outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)# The call to `model` always returns a tuple, so we need to pull the # loss value out of the tuple. loss = outputs[0] total_loss += loss.item()# Perform a backward pass to calculate the gradients. loss.backward()# Clip the norm of the gradients to 1.0. # This is to help prevent the "exploding gradients" problem. torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() scheduler.step()# Calculate the average loss over the training data. avg_train_loss = total_loss / len(train_dataloader)# Store the loss value for plotting the learning curve. loss_values.append(avg_train_loss)print("") print("Average training loss: {0:.2f}".format(avg_train_loss)) print("Training epcoh took: {:}".format(format_time(time.time() - t0)))# ======================================== #Validation # ======================================== # After the completion of each training epoch, measure our performance on # our validation set.print("") print("Running Validation...")t0 = time.time()# Put the model in evaluation mode--the dropout layers behave differently # during evaluation. model.eval()# Tracking variables eval_loss, eval_accuracy = 0, 0 nb_eval_steps, nb_eval_examples = 0, 0# Evaluate data for one epoch for batch in validation_dataloader: # Add batch to GPU batch = tuple(t.to(device) for t in batch) b_input_ids, b_input_mask, b_labels = batch with torch.no_grad(): outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)# Get the "logits" output by the model. The "logits" are the output # values prior to applying an activation function like the softmax. logits = outputs[0]# Move logits and labels to CPU logits = logits.detach().cpu().numpy() label_ids = b_labels.to('cpu').numpy()# Calculate the accuracy for this batch of test sentences. tmp_eval_accuracy = flat_accuracy(logits, label_ids)# Accumulate the total accuracy. eval_accuracy += tmp_eval_accuracy# Track the number of batches nb_eval_steps += 1# Report the final accuracy for this validation run. print("Accuracy: {0:.2f}".format(eval_accuracy/nb_eval_steps)) print("Validation took: {:}".format(format_time(time.time() - t0)))print("") print("Training complete!")

训练结束后在测试集上进行测试

t0 = time.time() model.eval()# Tracking variables eval_loss, eval_accuracy = 0, 0 nb_eval_steps, nb_eval_examples = 0, 0# Evaluate data for one epoch for batch in test_dataloader:# Add batch to GPU batch = tuple(t.to(device) for t in batch)# Unpack the inputs from our dataloader b_input_ids, b_input_mask, b_labels = batch with torch.no_grad(): outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)# Get the "logits" output by the model. The "logits" are the output # values prior to applying an activation function like the softmax. logits = outputs[0]# Move logits and labels to CPU logits = logits.detach().cpu().numpy() label_ids = b_labels.to('cpu').numpy()# Calculate the accuracy for this batch of test sentences. tmp_eval_accuracy = flat_accuracy(logits, label_ids)# Accumulate the total accuracy. eval_accuracy += tmp_eval_accuracy# Track the number of batches nb_eval_steps += 1 print("Accuracy: {0:.4f}".format(eval_accuracy/nb_eval_steps)) print("Test took: {:}".format(format_time(time.time() - t0)))

最后我的结果