Depth Enhancement via Low-rank Matrix Completion

Si Lu1, Xiaofeng Ren2, and Feng Liu1

1Department of Computer Science, Portland State University 2Department of Computer Science and Engineering, University of Washington

Abstract

Depth captured by consumer RGB-D cameras is often noisy and misses values at some pixels, especially around object boundaries. Most existing methods complete the missing depth values guided by the corresponding color image. When the color image is noisy or the correlation between color and depth is weak, the depth map cannot be properly enhanced. In this paper, we present a depth map enhancement algorithm that performs depth map completion and de-noising simultaneously. Our method is based on the observation that similar RGB-D patches lie in a very low-dimensional subspace. We can then assemble the similar patches into a matrix and enforce this low-rank subspace constraint. This low-rank subspace constraint essentially captures the underlying structure in the RGB-D patches and enables robust depth enhancement against the noise or weak correlation between color and depth. Based on this subspace constraint, our method formulates depth map enhancement as a low-rank matrix completion problem. Since the rank of a matrix changes over matrices, we develop a data-driven method to automatically determine the rank number for each matrix. The experiments on both public benchmark and our own captured RGB-D images show that our method can effectively enhance depth maps.

Paper

Si Lu, Xiaofeng Ren, and Feng Liu. Depth Enhancement via Low-rank Matrix Completion.  IEEE CVPR 2014. PDF

Data

We apply our method on images from three datasets, namely the Middlebury dataset [1, 2, 3, 4], the RGBZ dataset [5] and our own RGBD dataset captured by ASUS Xtion Pro. You can download our data as a zip package or click the links below for details Middlebury dataset RGBZ dataset RGBD dataset

References

[1] H. Hirschmuller and D. Scharstein. Evaluation of cost functions for stereo matching. In IEEE CVPR, 2007. [2] D. Scharstein and C. Pal. Learning conditional random fields for stereo. In IEEE CVPR, 2007. [3] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision,      47(1-3):7–42, 2002. [4] D. Scharstein and R. Szeliski. High-accuracy stereo depth maps using structured light. In IEEE CVPR, 2003. [5] Christian Richardt, Carsten Stoll, Neil A. Dodgson, Hans-Peter Seidel, and Christian Theobalt. Coherent Spatiotemporal Filtering,       Upsampling and Rendering of RGBZ Videos. Comp. Graph. Forum, 31(2):247–256, 2012.