【python 走进NLP】从零开始搭建textCNN卷积神经网络模型

无意中发现了一个巨牛的人工智能教程,忍不住分享一下给大家。教程不仅是零基础,通俗易懂,而且非常风趣幽默,像看小说一样!觉得太牛了,所以分享给大家。点这里可以跳转到教程。人工智能教程

1、众所周知,tensorflow 是一个开源的机器学习框架,它的出现大大降低了机器学习的门槛,即使你没有太多的数学知识,它也可以允许你用“搭积木”的方式快速实现一个神经网络,即使没有调节太多的参数,模型的表现一般还不错。目前,tensorflow 的安装已经变得非常简单,一个简单的 pip install tensorflow 即可,然后 import tensorflow as tf 就能愉快玩耍了。

2、卷积神经网络,即CNN,它的核心思想是捕捉数据的局部特征。不仅仅在图像领域大放异彩,CNN在文本分类领域也有很强的表现。在Yoon Kim的这篇 论文 中,比较清楚地解释了CNN用于文本分类的原理,关键在于如何将文本向量化,如下图,即把每个词都表示为一个 1×k的向量,对长度为N的文本则表示为N×K的矩阵,经过这一步处理,那么我们就可以把图像上的分类经验应用到文本上来了。

def _read_file(txt_file):
    """读取txt文件"""
    return open(txt_file, 'rb').read().decode("gbk", 'ignore')

3、那么又如何将文本转换为向量呢。有两种方法,第一种,采用预训练模型训练好的词向量。第二种呢,就是从原始文本中建立词汇表,然后把文本中的每个字符都对应编码。比如,“我爱北京天安门。”,我们就会把这段文本全部打散成为“我”、“爱”、“北”、“京”、“天”、“安”、“门”、“。”,甚至标点符号、特殊字符都会有对应的编码,不过从模型的表现来看,效果很好。

4、构建词汇表。筛选出的训练集语料中出现频次较高的5000个字符作为词汇表,我比较好奇的是我并没有对原始语料做任何的清洗、去噪,却丝毫不影响分类器的表现。添加一个 来将所有文本pad为同一长度

build_vocab(): 构建词汇表,使用字符级的表示,这一函数会将词汇表存储下来,避免每一次重复处理;

def build_vocab(train_path, vocab_path, vocab_size=5000):
    """构建词汇表"""
    data_train, _ = read_file(train_path)

    all_data = []
    for content in data_train:
        all_data.extend(content)

    counter = Counter(all_data)
    counter_pairs = counter.most_common(vocab_size-1)
    words, _ = list(zip(*counter_pairs))
    words = ['<PAD>'] + list(words)
    open_file(vocab_path, mode='w').write('\n'.join(words) + '\n')

5、词汇表建立好了,txt文件并不适合查询,所以这里用字符在文件的顺序作为其标识的id,存储到字典 word_to_id 中,这样以来就方便查找了。
read_vocab(): 读取上一步存储的词汇表,转换为{词:id}表示;

def read_vocab(vocab_path):
    with open(vocab_path) as f:
        words = [_.strip() for _ in f.readlines()]
        word_to_id = dict(zip(words, range(len(words))))
    return words, word_to_id

6、因变量编码
read_category(): 将分类目录固定,转换为{类别: id}表示;

def read_category():
    categories = ['mil.news', 'cul', 'health', 'travel', 'auto', 'learning', 'it', 'yule', 'sports', 'business', 'news']
    cat_to_id = dict(zip(categories, range(len(categories))))
    return categories, cat_to_id

7、处理数据。做完构建词汇表、类别转换为one-hot编码的准备工作,终于要进入正题了,数据进入模型训练、验证、测试前的准备工作还没有做。下面, process_file() 函数首先读取数据文件,将正文和标签分别对应存储在 contents 和 labels 两个列表中,然后再处理 contents 中的每一段文本,把文本中每一个字符在词汇表中找到其对应的id,完成文本数值化操作。类别转换为one-hot表示: y_pad = kr.utils.to_categorical(label_id, num_classes=len(cat_to_id)) 。

process_file(): 将数据集从文字转换为固定长度的id序列表示;

def process_file(file_name, word_to_id, cat_to_id, max_length=600):
    contents, labels = read_file(file_name)

    data_id, label_id = [], []
    for i in range(len(contents)):
        data_id.append([word_to_id[x] for x in contents[i] if x in word_to_id])
        label_id.append(cat_to_id[labels[i]])

    # 使用keras提供的pad_sequences来将文本pad为固定长度
    x_pad = kr.preprocessing.sequence.pad_sequences(data_id, max_length)
    y_pad = kr.utils.to_categorical(label_id, num_classes=len(cat_to_id))  # 将标签转换为one-hot表示

    return x_pad, y_pad

batch_iter(): 为神经网络的训练准备经过shuffle的批次的数据。

def batch_iter(x, y, batch_size=64):
    """生成批次数据"""
    data_len = len(x)
    num_batch = int((data_len - 1) / batch_size) + 1

    indices = np.random.permutation(np.arange(data_len))
    x_shuffle = x[indices]
    y_shuffle = y[indices]

    for i in range(num_batch):
        start_id = i * batch_size
        end_id = min((i + 1) * batch_size, data_len)
        yield x_shuffle[start_id:end_id], y_shuffle[start_id:end_id]

8、CNN配置

class TCNNConfig(object):
    """CNN配置参数"""

    embedding_dim = 64  # 词向量宽度
    seq_length = 1000  # 输入矩阵的宽度
    num_classes = 11  # 类别数
    num_filters = 256  # 卷积核数目
    kernel_size = 5  # 卷积核尺寸,即卷积核覆盖的词汇数量
    vocab_size = 6000  # 词汇表大小

    hidden_dim = 128  # 全连接层神经元

    dropout_keep_prob = 0.5  # dropout保留比例
    learning_rate = 1e-3  # 学习率

    batch_size = 64  # 每批训练大小
    num_epochs = 10  # 总迭代轮次

    print_per_batch = 100  # 每多少轮输出一次结果
    save_per_batch = 10  # 每多少轮存入tensorboard

9、CNN模型

class TextCNN(object):
    """文本分类,CNN模型"""

    def __init__(self, config):
        self.config = config

        # 三个待输入的数据,腾出占位符
        # input_x 为 n * seq_length 的矩阵,n 大小不固定
        # input_y 同
        self.input_x = tf.placeholder(tf.int32, [None, self.config.seq_length], name='input_x')
        self.input_y = tf.placeholder(tf.float32, [None, self.config.num_classes], name='input_y')
        self.keep_prob = tf.placeholder(tf.float32, name='keep_prob')

        self.cnn()

    def cnn(self):
        """CNN模型"""
        # 词向量映射
        with tf.device('/cpu:0'): # 强制使用CPU
            embedding = tf.get_variable('embedding', [self.config.vocab_size, self.config.embedding_dim])
            embedding_inputs = tf.nn.embedding_lookup(embedding, self.input_x)

        with tf.name_scope("cnn"):
            # CNN layer
            conv = tf.layers.conv1d(embedding_inputs, self.config.num_filters, self.config.kernel_size, name='conv')
            # global max pooling layer
            gmp = tf.reduce_max(conv, reduction_indices=[1], name='gmp')

        with tf.name_scope("score"):
            # 全连接层,后面接dropout以及relu激活
            # 激活函数后得到第二个全连接层
            fc = tf.layers.dense(gmp, self.config.hidden_dim, name='fc1')
            fc = tf.contrib.layers.dropout(fc, self.keep_prob)
            fc = tf.nn.relu(fc) # 修正线性单元激活函数,大于零才被激活

            # 分类器
            self.logits = tf.layers.dense(fc, self.config.num_classes, name='fc2')
            self.y_pred_cls = tf.argmax(tf.nn.softmax(self.logits), 1)  # 预测类别,返回最大值的索引

        with tf.name_scope("optimize"):
            # 损失函数,交叉熵	
            cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(logits=self.logits, labels=self.input_y)
            self.loss = tf.reduce_mean(cross_entropy)
            # 优化器
            self.optim = tf.train.AdamOptimizer(learning_rate=self.config.learning_rate).minimize(self.loss)

        with tf.name_scope("accuracy"):
            # 准确率
            correct_pred = tf.equal(tf.argmax(self.input_y, 1), self.y_pred_cls)
            self.acc = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

10、train训练

def train():
    print("Configuring TensorBoard and Saver...")
    # 配置 Tensorboard,重新训练时,请将tensorboard文件夹删除,不然图会覆盖
    tensorboard_dir = 'tensorboard/textcnn'
    if not os.path.exists(tensorboard_dir):
        os.makedirs(tensorboard_dir)

    tf.summary.scalar("loss", model.loss)
    tf.summary.scalar("accuracy", model.acc)
    merged_summary = tf.summary.merge_all()
    writer = tf.summary.FileWriter(tensorboard_dir)

    # 配置 Saver
    saver = tf.train.Saver()
    if not os.path.exists(save_dir):
        os.makedirs(save_dir)

    print("Loading training and validation data...")
    # 载入训练集与验证集
    start_time = time.time()
    x_train, y_train = process_file(train_dir, word_to_id, cat_to_id, config.seq_length)
    x_val, y_val = process_file(val_dir, word_to_id, cat_to_id, config.seq_length)
    time_dif = get_time_dif(start_time)
    print("Time usage:", time_dif)

    # 创建session
    session = tf.Session()
    session.run(tf.global_variables_initializer())
    writer.add_graph(session.graph)

    print('Training and evaluating...')
    start_time = time.time()
    total_batch = 0  # 总批次
    best_acc_val = 0.0  # 最佳验证集准确率
    last_improved = 0  # 记录上一次提升批次
    require_improvement = 1000  # 如果超过1000轮未提升,提前结束训练

    flag = False
    for epoch in range(config.num_epochs):
        print('Epoch:', epoch + 1)
        batch_train = batch_iter(x_train, y_train, config.batch_size)
        for x_batch, y_batch in batch_train:
            feed_dict = feed_data(x_batch, y_batch, config.dropout_keep_prob)

            if total_batch % config.save_per_batch == 0:
                # 每多少轮次将训练结果写入tensorboard scalar
                s = session.run(merged_summary, feed_dict=feed_dict)
                writer.add_summary(s, total_batch)

            if total_batch % config.print_per_batch == 0:
                # 每多少轮次输出在训练集和验证集上的性能
                feed_dict[model.keep_prob] = 1.0
                loss_train, acc_train = session.run([model.loss, model.acc], feed_dict=feed_dict)
                loss_val, acc_val = evaluate(session, x_val, y_val)  # todo

                if acc_val > best_acc_val:
                    # 保存最好结果
                    best_acc_val = acc_val
                    last_improved = total_batch
                    saver.save(sess=session, save_path=save_path)
                    improved_str = '*'
                else:
                    improved_str = ''

                time_dif = get_time_dif(start_time)
                msg = 'Iter: {0:>6}, Train Loss: {1:>6.2}, Train Acc: {2:>7.2%},' \
                      + ' Val Loss: {3:>6.2}, Val Acc: {4:>7.2%}, Time: {5} {6}'
                print(msg.format(total_batch, loss_train, acc_train, loss_val, acc_val, time_dif, improved_str))

            session.run(model.optim, feed_dict=feed_dict)  # 运行优化
            total_batch += 1

            if total_batch - last_improved > require_improvement:
                # 验证集正确率长期不提升,提前结束训练
                print("No optimization for a long time, auto-stopping...")
                flag = True
                break  # 跳出循环
        if flag:  # 同上
            break


11、测试test函数

def test():
    print("Loading test data...")
    start_time = time.time()
    x_test, y_test = process_file(test_dir, word_to_id, cat_to_id, config.seq_length)

    session = tf.Session()
    session.run(tf.global_variables_initializer())
    saver = tf.train.Saver()
    saver.restore(sess=session, save_path=save_path)  # 读取保存的模型

    print('Testing...')
    loss_test, acc_test = evaluate(session, x_test, y_test)
    msg = 'Test Loss: {0:>6.2}, Test Acc: {1:>7.2%}'
    print(msg.format(loss_test, acc_test))

    batch_size = 128
    data_len = len(x_test)
    num_batch = int((data_len - 1) / batch_size) + 1

    y_test_cls = np.argmax(y_test, 1)
    y_pred_cls = np.zeros(shape=len(x_test), dtype=np.int32)  # 保存预测结果
    for i in range(num_batch):  # 逐批次处理
        start_id = i * batch_size
        end_id = min((i + 1) * batch_size, data_len)
        feed_dict = {
            model.input_x: x_test[start_id:end_id],
            model.keep_prob: 1.0
        }
        y_pred_cls[start_id:end_id] = session.run(model.y_pred_cls, feed_dict=feed_dict)

    # 评估
    print("Precision, Recall and F1-Score...")
    print(metrics.classification_report(y_test_cls, y_pred_cls, target_names=categories))

    # 混淆矩阵
    print("Confusion Matrix...")
    cm = metrics.confusion_matrix(y_test_cls, y_pred_cls)
    print(cm)

    time_dif = get_time_dif(start_time)
    print("Time usage:", time_dif)

12、模型结构

在这里插入图片描述

demo:github项目学习地址:
https://github.com/gaussic/text-classification-cnn-rnn

已标记关键词 清除标记
©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页