Homework 3: Video Frame Interpolation

 Frame interpolation example. The source images are obtained from the Sintel benchmark.
The high-frame-rate movies begin to attract interest these recent years. More and more displays are able to play high-speed videos. Most of existing movies, however, are shot at 24 or 30 fps. The goal of this homework is to develop a video frame interpolation tool that can convert a 24 or 30 fps video to a 48 or 60 fps one. Specifically, we need to interpolate one frame between any two consecutive frames in the original low fps video.

Many algorithms are available for video frame interpolation now [1, 2, 3, 4, 5]. The Middlebury benchmark lists over 160 methods. The basic idea is to estimate the optical flow between two consecutive frames and generate the middle frame guided by the flow. You are asked to implement the interpolation algorithm described in Section 3.3 of [4]. We provide a skeleton code for this homework. We also provide some examples with the reference interpolation results from [4] and the ground truth. You can find these files here.

References:
[1] Manuel Werlberger, Thomas Pock, Markus Unger, and Horst Bischof. Optical Flow Guided TV-L1 Video Interpolation and Restoration
     EMMCVPR 2011
[2] Evan Herbst, Steve Seitz, Simon Baker. Occlusion reasoning for temporal interpolation using optical flow. Tech. report UW-CSE09-08-01
[3] Takaaki Shiratori, Yasuyuki Matsushita, Xiaoou Tang, and Sing Bing Kang. Video Completion by Motion Field Transfer. CVPR 06
[4] Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. A Database and Evaluation Methodology
     for Optical Flow, International Journal of Computer Vision, 92(1):1-31, March 2011.
[5] Simon Niklaus, Feng Liu. Context-aware frame synthesis for video frame interpolation. CVPR 2018.

Grading policy:
This homework has 100 points in total and will contribute 25% to your final grade of this course. You need to implement the algorithms for the functions in the skeleton file and we will grade each function individually. The details can be found in this slide presentation.

Submission:

The homework is due 4:30 pm, March 12. No late submission will be accepted. You should submit "frame_interpolation.py" to lizhan@pdx.edu. No other files should be submitted.