Content-Preserving Warps for 3D Video Stabilization

Feng Liu1, Michael Gleicher1, Hailin Jin2 and Aseem Agarwala2

1Computer Sciences Department, University of Wisconsin-Madison

2Adobe Systems Inc.

Abstract

In this paper, we describe a technique that transforms a video from a hand-held video camera so that it appears as if it were taken with a directed camera motion. Our method can adjust the video to appear as if it were taken from nearby viewpoints, allowing for 3D camera movements to be simulated. By aiming only for perceptual plausibility, rather than accurate reconstruction, we are able to develop algorithms that can effectively recreate dynamic scenes from a single source video. Our technique first recovers the original 3D camera motion and a sparse set of 3D, static scene points using an off-the-shelf structure-from-motion system. Then, a desired camera path is computed either automatically (e.g., by fitting a linear or quadratic path) or interactively. Finally, our technique performs a least-squares optimization that computes a spatially-varying warp from each input video frame into an output frame. The warp is computed to both follow the sparse displacements suggested by the recovered 3D structure, and avoid deforming the content in the video frame. Our experiments on stabilizing challenging videos of dynamic scenes demonstrate the effectiveness of our technique.

Paper
Feng Liu, Michael Gleicher, Hailin Jin and Aseem Agarwala. Content-Preserving Warps for 3D Video Stabilization
ACM Transactions on Graphics (Proceedings of SIGGRAPH 2009), 2009.  PDF:  big (9.0M ) small (1.2M)  
Related Projects
Feng Liu, Yuzhen Niu and Hailin Jin. Joint Subspace Stabilization for Stereoscopic Video
IEEE ICCV 2013. PDF  
Website
Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin and Aseem Agarwala. Subspace Video Stabilization
ACM Transactions on Graphics. Vol. 30, Issue 1, 2011: 4:1-4:10. PDF  
Website
Yu-Shuen Wang, Feng Liu, Pu-Sheng Hsu, and Tong-Yee Lee. Spatially and Temporally Optimized Video Stabilization.
IEEE Transactions on Visualization and Computer Graphics, 2013. Video Website
Michael Gleicher and Feng Liu. Re-cinematography: Improving the Camerawork of Casual Video. ACM Transactions on Multimedia Computing, Communications, and Applications. Vol. 5, Issue 1, Oct. 2008:1-28. PDF Website
Slides 
PDF  (Need Acrobat Reader 9.0 or later version to watch the embedded videos) 
SIGGRAPH Video 
Download        

Watch it on Youtube  or below   

Search engine friendly content

Video Figures 

Our paper refers to a number of "video figures" that accompany the text of our paper.  
These video figures are organized in the table below.

Figure number Description Streaming Download
1 Input video video video
2 2D stabilization result by weak low-pass filtering video video
3 2D stabilization result by strong low-pass filtering video video
4 3D stabilization result using unstructured lumigraph rendering video video
5 3D stabilization result using scattered data interpolation video video
6 3D stabilization result by applying a best-fit homography from the sparse scene
point correspondences
 
video video
7 3D stabilization result by applying a best-fit homography from the sparse scene
point correspondences  
video video
8 3D stabilization result using content-preserving warps (our result) video video
9 Our result with ease-in- and-out   video video
10 Our result without ease-in- and-out   video video
11 Our result without pre-warping video video
12 Our result with pre-warping video video
13 Our result using interactive camera path planning video video

Additional Videos 

 

Example 1

Input video

Our high-resolution result

Ours vs. iMovie

Ours vs. Deshaker

Ours vs. Our implementation of 2D Stabilization

Example 2

Input video

Our high-resolution result

Ours vs. iMovie

Ours vs. Deshaker

Ours vs. Our implementation of 2D Stabilization

Example 3

Input video

Our high-resolution result

Ours vs. iMovie

Ours vs. Deshaker

Ours vs. Our implementation of 2D Stabilization

Example 4

Input video

Our high-resolution result

Ours vs. iMovie

Ours vs. Deshaker

Ours vs. Our implementation of 2D Stabilization

Previous Work 
Re-Cinematography: Improving the Camerawork of Casual Video