Style Transfer: Does Deep Learning Understand Art Deeply?
Friday 31st August, 2018
Machine learning, Style transfer

How deeply does style transfer understand the styles it transfers? Not very, I suspect. Consider analytical cubism. This was the early cubism of Braque and others, who composed paintings from fragmentary images, each showing the same object from a different viewpoint. These paintings have a distinctive texture:
A selection of analytical cubist paintings [ Image: Google Image Search for "analytical cubism" ]

According to Google's usage-rights tool, most of the analytical cubist paintings are not public domain, so I'm restricted in what I can show. As the search results above are practically unusable, I hope I'm OK in claiming "fair use" for them. But one that is public domain is Juan Legua by Juan Gris (1911):
The painting 'Juan Legua' by Juan Gris [ Image: From https://www.metmuseum.org/art/collection/search/490060, Metropolitan Museum of Art ] This demonstrates the multiplicity of viewpoints and how these get pieced together, while the smaller pictures give an impression of the overall texture thus created.

What has this to do with style transfer? Were I to ask a program of the kind described in my Style Transfer post to render a photo in the style of one of the above paintings, I am sure it would do so. But it would do so in the way someone would whose only experience with analytical cubism comes from one of those tiny pictures.

In other words, it would have a superficial model of the target style, namely that the distribution of colour and tone is fairly even, that the colours are not bright, that there are a lot of short dark lines, that these tend to be straight or slightly curved, that they tend also to be fairly evenly spaced, and so on. What it would not have is a more profound model of the analytical cubist's intentions, as stated, for example, in this quote by Peter Vergo:†

What the Cubists had done was to create a new image of reality, influenced to some extent by the radical theories of the French philosopher Henri Bergson. Rejecting any conception of painting as a kind of 'window on the world', they broke decisively with the post-Renaissance convention of depicting objects as if seen from a single viewpoint, employing instead what Metzinger called 'mobile perspective' — moving round objects, simultaneously recording not only different images of the same object, but also the near and the far, the seen and the remembered. The more radical also analysed, probed, destroyed objects in order to reconstruct them, enhancing the emphasis given to the surface plane of the picture while at the same time progressively blurring the separation between the motif (figure, object, etc.) and its environment.

Why does this matter? I don't know whether it does matter to style transfer for clothing design. But it's good for programs to have as deep an understanding as possible of their task, and this seems to be a case where style transfer is currently lacking. I'll give another example, probably more relevant to fashion, in a later post. Meanwhile, I'd like to issue a challenge to the style-transfer researchers: build a program that does cubism properly!

†From Peter Vergo's Introduction to Abstraction: Towards a New Art. Painting 1910-20, Tate Gallery, 1980