TIP and Style Transfer

Varun Neal Srivastava

Part 1: Tour into the Picture

This part follows the description of the paper Tour into the Picture by Horry et al.

Background

In order to project a single view photograph into 3-dimensions, we approximate the picture into five planes: the ground, the ceiling, the left and right walls, and the back wall. When a picture conforms easily to these assumptions, we can recover the relative location of each of these planes in order to create a 3D projection. The location of each of these planes on the photograph can be fully determined by the location of the back wall and the location of the vanishing point——where planar lines converge.

The location of each of these planes on the photograph can be fully determined by the location of the back wall and the location of the vanishing point——where planar lines converge. These can be hand-selected via a user interface.

Via the geometry of the pinhole model, the relative locations of the planes could almost be fully determined. The camera could be assumed to occupy a point in 3D space, say (0,1,0). The location of the image screen relative to the pinhole would need to be guessed.

The size of the image and distance from the pinhole (e.g. focal length) are geometrically the same. They impact the 3D depth of the image, and essentially must be guessed. The (x, y) offset of the image from the pinhole can be calculated so that parallel lines on the image are sent to parallel lines in 3D. In reality, this calculation is over-constrained by the six sets of parallel lines in the model. Rather than finding a least-squares fit, I calculated the offsets such that the ground plane was properly aligned. After all 3D coordinates were recovered, I took the average of points that should map to the same location to ensure that all planes were orthogonal and all parallel lines were parallel.

Below is a collection of results from the interior of 3D renders.

original

Part 2: Style Transfer

This part follows the description of the classic paper A Neural Algorithm of Artistic Style by Gatys et al.

Style Transfer Overview

The hypothesis of style transfer neural algorithm is that CNNs embed the "style" of images in their hidden layers. Therefore, if we diffuse/gradient descent on the pixels of an image in order to match the style of another image, we can achieve style transfer. This turns out to be very simple to implement. I used the pre-trained SqueezeNet model for its small size. It consists 13 CNN modules followed by a classifier. We derived "style" from a weighted average of the 1st, 4th, 6th, and 7th layers, whereas the "content" tensor was derived from the 3rd layer.

To measure content loss, SSD was performed, whereas for style loss, the correlation between features were measured via a Gram matrix. The SSD of Gram matrices was taken, rather than the SSD of the features directly. Finally, a sort of variation loss was also measured, which measure the variance of pixels over the output image. This was done in order to create some smoothness across the output. Naively, a weighted average of these losses was taken for each optimization step.

Finding the right hyperparameters was a guessing game that took several tries.

But after I found them, I could turn anything into Starry Night.

Of course, I turned them into The Wave as well.

Combination

A wise man once said "I'm at the Pizza Hut. I'm at the Taco Bell. I'm at the combination Pizza Hut and Taco Bell." Here's the 3D render of a couple style transferred images:

Conclusion

To model my 3D images I used the very-simple vpython. For my Bells and Whistles, I created the python version of the pre-provided Matlab code (which I have since emailed to the relevant parties).

The most enjoyable part of the final project was definitely working out all of the math regarding the pinhole camera. Reading the original paper, which was rather terse, made me redo a lot of calculations by hand. Due to this, I have a much better understanding of the geometry of single-view images. I also see parallel lines and vanishing points around me everywhere now.