Part A: Image Warping and Mosaics

Defining Correspondences

Correspondences were manually selected. In order to help this process, I built a fine-tuning tool that would automatically "snap" a correspondence to a nearby pixel with a higher NCC. This resulted in fast and accurate correspondence-selection.

Rectification

Finding the keypoint correspondences between two planar surfaces allows us to compute the homography matrix H that transforms one planar surface to the other.
One application of this is to "rectify"—front align—a given image. Below is such an example on my pencil box.

Original image of my pencil box.

Rectified with top of pencil box facing camera.

This works quite well. It is worth noting an interpolation is needed in order to complete the warp.
Below is another example of rectification on Holbein's 1533 Painting The Ambassadors. Particularly, we are rectifying the skull present in the bottom left of the image.

The Ambassadors

Cropped to skull, with bounding box for reference.

Cropped rectification of the skull.

We are having a lot of fun rectifying! I am going to use this technique to make a meme.

Walt Jr. Showing his father his computer.

KID NAMED FINGER meme.



Mosaics

We have the tools to stitch together mosaics!
We will use a distance transform as our mask — this will allow us to determine the boundaries for our images, and the behavior of the mosaic on the regions where two images overlap.

Left

Distance transform of Left

Right

Distance transform of Right

Binary mask

Final image

Oh no! The lighting is all fucked up. Instead of using a binary mask, let's apply a Gaussian Blur to the mask to smooth over transitions.

Blurred mask

New final image

Much better! Let's use this technique to make more mosaics:

bottom

middle

top

full

Anthropology building:

left

middle

right

full (slightly cropped)

It is clear in the above example that it matters which perspective we choose to align each image. In the above example, the first and second images are greatly distorted to match the plane of the third image.

In an attempt to do some fun stuff, I tried to make myself floating. Unfortunately, the lighting wasn't great here, and there were not many good corespondences, but the strange warping gave me the added effect of looking taller!

bottom

top

floating and tall? dream man



Part A Interlude

It has been incredibly fun to start thinking of images in terms of the projective planes that they lay on. In order to implement the homography matrix, I spent a lot of time playing with matrices by-hand to see how they would map planes (parallelograms) to each other.

Part B: Automosaics

Feature Selection

We use Harris Corners and the ANMS refinement technique to identify interest points automatically.

Original image

Harris Corners —— way too many features!

ANMS Corners (much fewer)



Feature Descriptions

Next we extract feature vectors around each interest point, which consist of a rescaled, blurred 40 x 40 vector around the point, resulting in a 64-dimensional descriptor. Below are some example description vectors.



Matching Features

Now we have to match features to derive correspondences. We do this by:

  1. Finding Nearest Neighbors between interest points
  2. Using Lowe Thresholding to select for the "best" neighbors
  3. Using 4-RANSAC to refine our selection
In particular, the 4-RANSAC technique was the same as that described in class; 4-point homographies were randomly sampled and a sufficiently good homography's inliers were used as the final set of correspondences. In the background is an example of these automatically generated correspondences. They look pretty accurate!.

Results

Now we can use these correspondences to compute our mosaics. Here are comparisons of some of our "by-hand" panoramics and corresponding automosaics:

Hand-selected correspondences

Automaticly selected correspondences

Hand-selected correspondences

Automaticly selected correspondences

Hand-selected correspondences

Automaticly selected correspondences



Bells and Whistles: Panorama Recognition

Given a set of unordered images, we attempt to recognize which ones form a panorama, and then proceed to stitch those images together into a panorama. The technique I used for this was naive: if two images have many correspondence points, they can be stitched together. Putting together all the images that can be stitched together (with some consideration to order) will form a panorama.
I took 20 random-ish images (pictured below with similar images placed next to each other), a few of which depicted the same object from multiple perspectives, and algorithmically extracted the mosaics that could be turned into panorammics.

This was successful and yielded the following panoramas:


The last panoramic (of my parents' wedding) was not actually of the same image, so it makes sense that the resulting panoramic looks incorrect. That said, it makes sense that they were detected as potential candidates for a panoramic and the algorithm attempted to match them.



Conclusion

One of the coolest techniques I learned from this project was Lowe Thresholding. I am surprised at how effective it is as a metric for quality selection, as it seems to be empirically very reliable. I am also surprised that a naive nearest neighbors worked, without implementing rotational invariance, on my feature vectors. I think that this speaks to the power of simple metrics being sufficiently effective much of the time.