Content & Style Losses

The problem, from first principles

Style transfer is optimization over the pixels of a new image, not over network weights. The pretrained CNN is frozen and the image is the parameter. The objective has two parts. The content loss keeps the generated image's feature maps close to those of the content photo at a deep layer, so the objects and layout are preserved. The style loss keeps the generated image's Gram matrices close to those of the style painting at several layers, so textures and colours match. A weighted sum of the two (plus a smoothness term, covered next) is minimized by gradient descent on the image. The weights express the trade-off between "looks like the photo" and "looks like the painting".

From theory to code

Implement content_loss, style_loss and total_loss.

Constraints

• content_loss(gen_feat, content_feat) is the mean squared error between two feature arrays of the same shape.
• style_loss(gen_feats, style_feats, layer_weights): lists of (C, H, W) feature arrays, one per style layer. For each layer compute the Gram matrix (F @ F.T / (C * H * W) with F = feat.reshape(C, -1)) and the mean squared error between the generated and style Grams. Return sum(w * mse) over layers.
• total_loss(content, style, tv, w_content, w_style, w_tv) returns w_content * content + w_style * style + w_tv * tv.

Hints