Notes - Image Style Transfer Using Convolutional Neural Networks

The algorithm separates and recombines image content and style of images.

The target image is the image desired to transfer a texture to. The general idea is to extract high level features like the objects and scenery then apply the texture procedure.
They start with a VGG network that was pretrained to do object recognition and localization.
Content Representation:
In order to visualize the features a layer in the network contains let Nl represent the number of feature maps and Ml be the height x width of each feature map.
So you have Nl is the number of filters and Ml is a 1D compression of the 2Dfilters
Fl is just the filters at a layer.
Propagate an image p and white noise image x through the network with the goal of making p look like x
The loss is the L2 norm of between the features generated by p and x at a layer.
Propagate the derivative of the loss wrt each filter back to x in order to update x.
Doing this you find that the higher level layers contain features the correspond to the actual content of the image instead of the precise appearance.
This indicates features extracted from high level layers are develop content representation
Style Representation:
This looks at correlations between filter weights by averaging feature maps.
It specifically calcualtes the Gram matrix which is the dot product between different filters in a layer at the same position of each filter.
This shows how different filters in a layer correlate with eachother.
To capture style the l2 norm between gram matricies generated between the originial image and a white noise image are calculated as the style loss.
This is propagated back to the white noise image.
Style Transfer:
The two steps above are combine where high level features from the original image are extracted from the fourth layer and the loss between the filter correlations at each layer are calculated and propagated to the originial image.
Each layer has a different weight that corresponds to the amount of style that is contributed.