样式转换的工作原理详细实例图解

本文概述

导入和配置模块
创建显示图像的功能
定义样式和内容表示
样式和内容的中间层
克矩阵
运行梯度下降
变换图像
总变化损失
重新运行优化功能
最终保存结果

神经样式转移是用于获取两个图像(内容图像和样式参考图像)并将它们融合在一起的优化技术, 因此输出图像看起来像内容图像, 但以样式参考图像的样式” 绘制” 。
导入和配置模块开启Google colab

from __future__ import absolute_import, division, print_function, unicode_literals

try:# %tensorflow_version only exists in Colab.%tensorflow_version 2.xexcept Exception:passimport tensorflow as tf

输出

TensorFlow 2.x selected.

import IPython.display as displayimport matplotlib.pyplot as pltimport matplotlib as mplmpl.rcParams['figure.figsize'] = (12, 12)mpl.rcParams['axes.grid'] = Falseimport numpy as npimport timeimport functoolscontent_path = tf.keras.utils.get_file('nature.jpg', 'https://www.eadegallery.co.nz/wp-content/uploads/2019/03/626a6823-af82-432a-8d3d-d8295b1a9aed-l.jpg')style_path = tf.keras.utils.get_file('cloud.jpg', 'https://i.pinimg.com/originals/11/91/4f/11914f29c6d3e9828cc5f5c2fd64cfdc.jpg')

输出

Downloading data from https://www.eadegallery.co.nz/wp-content/uploads/2019/03/626a6823-af82-432a-8d3d-d8295b1a9aed-l.jpg1122304/1117520 [==============================] - 1s 1us/stepDownloading data from https://i.pinimg.com/originals/11/91/4f/11914f29c6d3e9828cc5f5c2fd64cfdc.jpg49152/43511 [=================================] - 0s 0us/step5. def

检查最大尺寸为512像素。

load_img(path_to_img):max_dim = 512img = tf.io.read_file(path_to_img)img = tf.image.decode_image(img, channels=3)img = tf.image.convert_image_dtype(img, tf.float32)shape = tf.cast(tf.shape(img)[:-1], tf.float32)long_dim = max(shape)scale = max_dim / long_dimnew_shape = tf.cast(shape * scale, tf.int32)img = tf.image.resize(img, new_shape)img = img[tf.newaxis, :]return img

创建显示图像的功能

def imshow(image, title=None): if len(image.shape) > 3: image = tf.squeeze(image, axis=0)plt.imshow(image)if title:plt.title(title)

content_image = load_img(content_path)style_image = load_img(style_path)plt.subplot(1, 2, 1)imshow(content_image, 'Content Image')plt.subplot(1, 2, 2)imshow(style_image, 'Style Image')

输出

文章图片

x = tf.keras.applications.vgg19.preprocess_input(content_image*255)x = tf.image.resize(x, (224, 224))vgg = tf.keras.applications.VGG19(include_top=True, weights='imagenet')prediction_probabilities = vgg(x)prediction_probabilities.shape

输出

Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg19_weights_tf_dim_ordering_tf_kernels.h5574717952/574710816 [==============================] - 8s 0us/stepTensorShape([1, 1000])

predicted_top_5 = tf.keras.applications.vgg19.decode_predictions(prediction_probabilities.numpy())[0][(class_name, prob) for (number, class_name, prob) in predicted_top_5]

输出

Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/imagenet_class_index.json40960/35363 [==================================] - 0s 0us/step[('mobile_home', 0.7314594), ('picket_fence', 0.119986326), ('greenhouse', 0.026051044), ('thatch', 0.023595566), ('boathouse', 0.014751049)]

定义样式和内容表示使用模型的中间层来表示图像的内容和样式。从输入层开始, 前几层激活表示低级表示类似的边缘和纹理。
对于输入图像, 请尝试在中间层匹配相似的样式和内容目标表示。
加载VGG19并在我们的映像上运行, 以确保在此处正确使用它。

vgg = tf.keras.applications.VGG19(include_top=False, weights='imagenet')print()for layer in vgg.layers:print(layer.name)

输出

Download data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg19_weights_tf_dim_ordering_tf_kernels_notop.h580142336/80134624 [==============================] - 2s 0us/stepinput_2block1_conv1block1_conv2block1_poolblock2_conv1block2_conv2block2_poolblock3_conv1block3_conv2block3_conv3block3_conv4block3_poolblock4_conv1block4_conv2block4_conv3block4_conv4block4_poolblock5_conv1block5_conv2block5_conv3block5_conv4block5_pool

# Content layercontent_layers = ['block5_conv2'] # Style layer of intereststyle_layers = ['block1_conv1', 'block2_conv1', 'block3_conv1', 'block4_conv1', 'block5_conv1']num_content_layers = len(content_layers)num_style_layers = len(style_layers)

样式和内容的中间层在较高的层次上, 对于执行图像分类的网络, 它了解图像并要求将图像作为像素并构建内部插图, 该插图将原始图像像素转换为图像中存在的复杂特征。
这也是卷积神经网络可以很好地泛化的原因：它们可以捕获类内的偏差和定义特征(例如猫与狗), 这些特征与图像输入模型和输出排列标签, 模型无关作为复杂的特征提取器提供。通过访问模型的中间层, 我们可以描述输入图像的样式和内容。
建立模型
定义了tf.keras.applications中的网络, 因此我们可以使用Keras功能API轻松提取中间层值。
要使用功能性API定义任何模型, 请指定输入和输出：
模型=模型(输入, 输出)
给定的函数将构建一个VGG19模型, 该模型返回中间层的列表。

def vgg_layers(layer_names):""" Creating a vgg model that returns a list of intermediate output values."""# Load our model. Load pretrained VGG, trained on imagenet datavgg = tf.keras.applications.VGG19(include_top=False, weights='imagenet')vgg.trainable = Falseoutputs = [vgg.get_layer(name).output for name in layer_names]model = tf.keras.Model([vgg.input], outputs)return model

style_extractor = vgg_layers(style_layers)style_outputs = style_extractor(style_image*255)#Look at the statistics of each layer's outputfor name, output in zip(style_layers, style_outputs):print(name)print("shape: ", output.numpy().shape)print("min: ", output.numpy().min())print("max: ", output.numpy().max())print("mean: ", output.numpy().mean())print()

输出

block1_conv1shape:(1, 427, 512, 64)min:0.0max:763.51953mean:25.987665block2_conv1shape:(1, 213, 256, 128)min:0.0max:3484.3037mean:134.27835block3_conv1shape:(1, 106, 128, 256)min:0.0max:7291.078mean:143.77878block4_conv1shape:(1, 53, 64, 512)min:0.0max:13492.799mean:530.00244block5_conv1shape:(1, 26, 32, 512)min:0.0max:2881.529mean:40.596397

克矩阵计算风格
图像的内容由地图共同特征的值表示。
通过在所有位置获取输出乘积来计算包括此信息的克矩阵。
可以针对特定层计算Gram矩阵, 如下所示：

文章图片
这是使用tf.linalg.einsum函数简洁地实现的：

def gram_matrix(input_tensor):result = tf.linalg.einsum('bijc, bijd-> bcd', input_tensor, input_tensor)input_shape = tf.shape(input_tensor)num_locations = tf.cast(input_shape[1]*input_shape[2], tf.float32)return result/(num_locations)

提取图像的样式和内容
构建返回内容和样式张量的模型。

class StyleContentModel(tf.keras.models.Model):def __init__(self, style_layers, content_layers):super(StyleContentModel, self).__init__()self.vgg =vgg_layers(style_layers + content_layers)self.style_layers = style_layersself.content_layers = content_layersself.num_style_layers = len(style_layers)self.vgg.trainable = Falsedef call(self, inputs):"Expects float input in [0, 1]"inputs = inputs*255.0preprocessed_input = tf.keras.applications.vgg19.preprocess_input(inputs)outputs = self.vgg(preprocessed_input)style_outputs, content_outputs = (outputs[:self.num_style_layers], outputs[self.num_style_layers:])style_outputs = [gram_matrix(style_output) for style_output in style_outputs]content_dict = {content_name:value for content_name, value in zip(self.content_layers, content_outputs)}style_dict = {style_name:valuefor style_name, valuein zip(self.style_layers, style_outputs)}return {'content':content_dict, 'style':style_dict}

当在图像上调用时, 此模型返回style_layers的语法矩阵(样式)和content_layers的内容：

extractor = StyleContentModel(style_layers, content_layers)results = extractor(tf.constant(content_image))style_results = results['style']print('Styles:')for name, output in sorted(results['style'].items()):print("", name)print("shape: ", output.numpy().shape)print("min: ", output.numpy().min())print("max: ", output.numpy().max())print("mean: ", output.numpy().mean())print()print("Contents:")for name, output in sorted(results['content'].items()):print("", name)print("shape: ", output.numpy().shape)print("min: ", output.numpy().min())print("max: ", output.numpy().max())print("mean: ", output.numpy().mean())

输出

Styles:block1_conv1shape:(1, 64, 64)min:0.0055228453max:28014.557mean:263.79025block2_conv1shape:(1, 128, 128)min:0.0max:61479.496mean:9100.949block3_conv1shape:(1, 256, 256)min:0.0max:545623.44mean:7660.976block4_conv1shape:(1, 512, 512)min:0.0max:4320502.0mean:134288.84block5_conv1shape:(1, 512, 512)min:0.0max:110005.37mean:1487.0381Contents:block5_conv2shape:(1, 26, 32, 512)min:0.0max:2410.8796mean:13.764149

运行梯度下降使用此样式和内容提取器, 我们实现了样式转移算法。为此, 请评估图像输出中相对于每个目标的均方误差, 然后对损失进行加权求和。
设置我们的样式和内容目标值：

style_targets = extractor(style_image)['style']content_targets = extractor(content_image)['content']

定义一个tf.Variable包含要保留的图像。借助内容图像进行初始化(tf.Variable与内容图像具有相同的形状)：

image = tf.Variable(content_image)

这是一个浮动图像, 定义一个函数以将像素值保持在0到1之间：

def clip_0_1(image):return tf.clip_by_value(image, clip_value_min=0.0, clip_value_max=1.0)

创建优化器。该论文推荐LBFGS：

opt = tf.optimizers.Adam(learning_rate=0.02, beta_1=0.99, epsilon=1e-1)

要对其进行优化, 请使用两个损失的权重组合来获得总损失：

style_weight=1e-2content_weight=1e4

def style_content_loss(outputs):style_outputs = outputs['style']content_outputs = outputs['content']style_loss = tf.add_n([tf.reduce_mean((style_outputs[name]-style_targets[name])**2) for name in style_outputs.keys()])style_loss *= style_weight / num_style_layerscontent_loss = tf.add_n([tf.reduce_mean((content_outputs[name]-content_targets[name])**2) for name in content_outputs.keys()])content_loss *= content_weight / num_content_layersloss = style_loss + content_lossreturn loss

使用函数tf.GradientTape更新图像。

@tf.function()def train_step(image):with tf.GradientTape() as tape: outputs = extractor(image) loss = style_content_loss(outputs)grad = tape.gradient(loss, image)opt.apply_gradients([(grad, image)])image.assign(clip_0_1(image))

运行以下步骤进行测试：

train_step (image)train_step (image)train_step (image)plt.imshow(image.read_value()[0])

输出

文章图片
变换图像在此步骤中执行更长的优化：

import timestart = time.time()epochs = 10steps_per_epoch = 100step = 0for n in range(epochs):for m in range(steps_per_epoch):step += 1train_step(image)print(".", end='')display.clear_output(wait=True)imshow(image.read_value())plt.title("Train step: {}".format(step))plt.show()end = time.time()print("Total time: {:.1f}".format(end-start))

输出

文章图片

文章图片
总变化损失

def high_pass_x_y(image):x_var = image[:, :, 1:, :] - image[:, :, :-1, :]y_var = image[:, 1:, :, :] - image[:, :-1, :, :]return x_var, y_var

x_deltas, y_deltas = high_pass_x_y(content_image)plt.figure(figsize=(14, 10))plt.subplot(2, 2, 1)imshow(clip_0_1(2*y_deltas+0.5), "Horizontal Deltas: Original")plt.subplot(2, 2, 2)imshow(clip_0_1(2*x_deltas+0.5), "Vertical Deltas: Original")x_deltas, y_deltas = high_pass_x_y(image)plt.subplot(2, 2, 3)imshow(clip_0_1(2*y_deltas+0.5), "Horizontal Deltas: Styled")plt.subplot(2, 2, 4)imshow(clip_0_1(2*x_deltas+0.5), "Vertical Deltas: Styled")

输出

文章图片

文章图片
这表明高频分量是如何增加的。
该高频分量是边缘检测器。在给定的示例中, 我们从边缘检测器获得了相同的输出：

plt.figure(figsize=(14, 10))sobel = tf.image.sobel_edges(content_image)plt.subplot(1, 2, 1)imshow(clip_0_1(sobel [..., 0]/4+0.5), "Horizontal Sobel-edges")plt.subplot(1, 2, 2)imshow(clip_0_1(sobel[..., 1]/4+0.5), "Vertical Sobel-edges")

输出

文章图片
与此相关的正则化损失是值的平方和：

def total_variation_loss(image):x_deltas, y_deltas = high_pass_x_y(image)return tf.reduce_sum(tf.abs(x_deltas)) + tf.reduce_sum(tf.abs(y_deltas))

total_variation_loss(image).numpy()

输出

99172.59

那证明了它的作用。但是不需要自己实现, 它包括一个标准实现：

tf.image.total_variation(image).numpy()

输出

array([99172.59], dtype=float32)

重新运行优化功能选择函数total_variation_loss的权重：

total_variation_weight=30

现在, train_step函数：

@tf.function()def train_step(image):with tf.GradientTape() as tape:outputs = extractor(image)loss = style_content_loss(outputs)loss += total_variation_weight*tf.image.total_variation(image)grad = tape.gradient(loss, image)opt.apply_gradients([(grad, image)])image.assign(clip_0_1(image))

重新初始化优化变量：

image = tf.Variable(content_image)

并运行优化：

import timestart = time.time()epochs = 10steps_per_epoch = 100step = 0for n in range(epochs):for m in range(steps_per_epoch):step += 1train_step(image)print(".", end='')display.clear_output(wait=True)display.display(tensor_to_image(image))print("Train step: {}".format(step))end = time.time()print("Total time: {:.1f}".format(end-start))

【样式转换的工作原理详细实例图解】输出

文章图片
最终保存结果

file_name = 'styletransfer.png'tensor_to_image(image). save(file_name)try: from google. colab import filesexcept ImportError:passelse:files.download(file_name)