Frame alignment, panorama stitching, video stitching using Homographies. More videos available on my YouTube channel
Homographies are sort of a plane to plane mapping. In the images on the left we could see that a square on frame 270 could be mapped to frame 450 using the homography. It seems that the mapping would make the picture slightly tilted to the right. To get the homography, first we need to map a set of pixels from frame 270 to frame 450. We could get this using various descriptors, the one we used is SIFT descriptors. For each descriptions we run it through RANSAC algorithms to get the inliers and ignore the outliers of the data. At the end of RANSAC we should get points that are mapped from one plane to another. Then we treat those pixels as data points and get the Homography as a linear system.
Also, we could make a panorama out of multiple frames using homography. E.g. given 5 key frames of a video (90th, 270th, 450th, 630th, 810th), We could pick the frame 450th as the reference plane. Then we could project all other frames to frame 450 by calculating the homography of that frame to the 450th frame. One thing we have to be careful is that, the further the frame is from the 450th frame, the less they share pixels, and that could result a bad homography matrix. In order to circumvent this, we could calculate a homography of, say frame 90th to frame 270th, then calculate the homography of frame 270th to frame 450th, and to get the homography of frame 90th to frame 450th we could just multiply those homographies together.
Since now we have the projected frames, we could analyze each pixel location and determine what the background color of that pixel is by picking the mean, median, or mode of the nonzero pixels.