Estimation of Optical Flow using Semantic Segmentation and AES Deep Neural Network

Document Type : Original Article

Authors

Abstract

The importance and demand of visual scene understanding have been increasing because of autonomous systems development. Optical flow is known as an important tool for scene understanding. Current optical flow methods present general assumptions and spatial homogeneous for spatial structure of flow. In fact, the optical flow in an image depends on object class and the type of object movement. The first assumption in many methods in this field is the brightness constancy during movements of pixels between frames. This assumption is proven to be inaccurate in general. In this paper, we use recent development of deep convolutional networks in semantic segmentation of static scenes to divide an image in to different objects and also depends on type of the object different movement patterns are defined. Next, estimation of the optical flow is performed by using deep neural network for initial image which has been semantically segmented. The proposed method provides minimum error in optical flow measures for KITTI-2015 database and results in more accurate segmentation compared to state-of-the-art methods for several natural videos.

Keywords

References

[1]     Revaud, J.; Weinzaepfel, P.; Harchaoui, Z., Schmid, C. “Epic Flow: Edge-Preserving Interpolation of Correspondences for Optical Flow”; IEEE Conf. Comput. Vision Pattern Recgn. 2015, 1164-1172.##
[2]     Sun, D., Roth, S., Black, M. J. “A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them”; Int. J. Comput. Vision 2014, 106, 115-137.##
[3]     Butler, D. J.; Wulff, J.; Stanley, G. B.; Black, M. J. “A Naturalistic Open Source Movie for Optical Flow Evaluation”; European Conf. Computer Vision 2012, 7577, 611-625.##
[4]     Geiger, A.; Lenz, P.; Stiller, C.; Urtasun, R. “Vision Meets Robotics: The KITTI Dataset”; Int. J. Robot. Res. 2013, 32, 1231-1237.##
[5]     Yamaguchi, K.; McAllester, D. A.; Urtasun, R. “Robust Monocular Epipolar Flow Estimation”; Proc. CVPR IEEE 2013, 1862–1869.##
[6]     Yamaguchi, K.; McAllester, D. A.; Urtasun, R. “Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation”; European Conf. Computer Vision 2014, 8693, 756-771.##
[7]     Lucas, B.; Kanade, T. “An Iterative Image Registration Technique with an Application to Stereo Vision (DARPA)”; Proc.  DARPA Image Understanding Workshop 1981, 121-130.##
[8]     Horn, B. K. P.; Schunk, B. G. “Determining Optical Flow”; Artif. Intell. Rev. 1981, 17, 185-203.##
[9]     Papenberg, N.; Bruhn, A.; Brox, T.; Didas, S.; Weickert, J. “Highly Accurate Optic Flow Computation with Theoretically Justified Warping”; Int. J. Comput. Vision 2006, 67, 141-158.##
[10]  Yang, H.; Lin, W.; Lu, J. “DAISY Filter Flow: A Generalized Discrete Approach to Dense Correspondences”; IEEE Conf. Comput. Vision Pattern Recgn. 2014.##
[11]  Bao, L.; Yang, Q.; Jin, H. “Fast Edge-Preserving Patch Match for Large Displacement Optical Flow”; IEEE Trans. Image Process. 2014, 23, 4996-5006.##
[12]  Menze, M.; Heipke, C.; Geiger, A. “Discrete Optimization for Optical Flow”; German Conf. Pattern Recogn. 2015, 9358, 16-28.##
[13]  Yang, J.; Li, H. “Dense, Accurate Optical Flow Estimation With Piecewise Parametric Model”; IEEE Conf. Comput. Vision Pattern Recgn. 2015, 1019-1027.##
[14]  Sun, D.; Liu, C.; Pfister, H. “Local Layering for Joint Motion Estimation and Occlusion Detection”; IEEE Conf. Comput. Vision Pattern Recgn. 2014, 1098-1105.##
[15]  Sevilla-Lara, L.; Sun, D.; Jampani, V.; Black, M. J. “Optical Flow with Semantic Segmentation and Localized Layers”; IEEE Conf. Comput. Vision Pattern Recgn. 2016, 3889-3898.##
[16]  Farsi H.; Behmadi, S. "Video Quality Improvement Using Local Channel Encoder and Mixed Predictor by Wavelet, Neural Network and Genetic Algorithm"; J. Adv. Defense Sci. Technol. 2018, 9, 449-459.##
[17]  Zbontar, J.; LeCun, Y. “Computing the Stereo Matching Cost with a Convolutional Neural Network”; IEEE Conf. Comput. Vision Pattern Recgn. 2015, 1592–1599.##
[18]  Luo, W.; Schwing, A. G.; Urtasun, R. “Efficient Deep Learning for Stereo Matching”; IEEE Conf. Comput. Vision Pattern Recgn. 2016, 5695–5703.##
[19]  Geiger, A.; Lenz, P.; Urtasun, R. “Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite”; IEEE Conf. Comput. Vision Pattern Recgn. 2012.##
[20]  Badrinarayanan, V.; Kendall, A.; Cipolla, R. “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation”; IEEE Trans. Pattern Anal. 2017, 39, 2481-2495.##
[21]  Chantas, C.; Gkamas, T.; Nikou, C. “Variational-Bayes Optical Flow”; Journal of Mathematical and Imaging Vision 2014, 50, 199-213.##
[22]  Brostow, G. J.; Fauqueur, J.; Cipolla, R. “Semantic Object Classes in Video: A High-Definition Ground Truth Database”; Pattern Recogn. Lett. 2009, 30, 88-97.##
[23]  Tan, Z.; Liu, B.; Yu, N. “PPEDNet: Pyramid Pooling Encoder-Decoder Network for Real-Time Semantic Segmentation”; Int. Conf. Image and Graphics 2017, 328-339.##
[24]  Everingham, M.; Eslami, S. M. A.; Van Gool, L.; Williams, C. K. I.; Winn, J.; Zisserman, A. “The Pascal Visual Object Classes Challenge: A Retrospective”; Int. J. Computer Vision 2015, 111, 98-136.##
[25]  Sharmin, N.; Brad, R. “Optimal Filter Estimation for Lucas-Kanade Optical Flow”; Sensors 2012, 12, 12694-12709.##
[26]  Sun, D.; Yang, X.; Liu, M. Y.; Kautz, Y. “PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume”; IEEE Conf. Comput. Vision Pattern Recgn. 2018.##
[27]  Shelhamer, E.; Long, J.; Darrell, T. “Fully Convolutional Networks for Semantic Segmentation”; IEEE Trans. Pattern Anal. 2017, 39, 640–651.##
[28]  Paszke, A.; Chaurasia, A.; Kim, S.; Culurciello, E. “ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation”; arXiv preprint arXiv: 1606.02147, 2016.##
[29]  Nanfack, G.; Elhassouny, E.; Thami, R. O. H. “Squeeze-SegNet: A New Fast Deep Convolutional Neural Network for Semantic Segmentation”; Tenth Int. Conf. Machine Vision, 2017.##
[30]  Simonyan, K.; Zisserman, A. “Very Deep Convolutional Networks for Large-Scale Image Recognition”; arXiv Preprint arXiv: 1409.1556, 2014.##
[31]  Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. “Deep Lab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”; IEEE Trans. Pattern Anal. 2018, 40, 834-848.##
[32]  Noh, H.; Hong, S.; Han, B. “Learning Deconvolution Network for Semantic Segmentation”; IEEE Int. Conf. Comput. Vision 2015, 1520-1528.##
[33]  Tighe, J.; Lazebnik, S. “Super Parsing: Scalable Nonparametric Image Parsing with Super Pixels”; European Conference on Computer Vision 2010, 352-365.##
[34]  Hu, Y.; Song, R.; Li., Y. “Efficient Coarse-to-fine Patch Match for Large Displacement Optical Flow”; IEEE Conf. Comput. Vision Pattern Recgn. 2016, 5704-5712.##
[35]  Hu, Y.; Li, Y.; Song, R. “Robust Interpolation of Correspondences for Large Displacement Optical Flow”; IEEE Conf. Comput. Vision Pattern Recgn. 2017, 4791–4799.##
[36]  Maurer, D.; Stoll, M.; Bruhn, A. “Order-Adaptive and Illumination-Aware Variational Optical Flow Refinement”; Proc. of the British Machine Vision Conference 2017.##
[37]  Maurer, D.; Bruhn, A. “ProFlow: Learning to Predict Optical Flow”; arXiv preprint arXiv:1806.00800. 2018.##
[38]  Hur, J.; Roth, S. “Mirror Flow: Exploiting Symmetries in Joint Optical Flow and Occlusion Estimation”; IEEE Conf. Comput. Vision Pattern Recgn. 2017, 312-321.##
[39]  Meister, S.; Hur, J.; Roth, S. “Unflow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss”; Proc. AAAI Conf. Artificial Intelligence 2018.##

Files

History

  • Receive Date: 15 January 2019
  • Revise Date: 28 May 2019
  • Accept Date: 02 June 2020
  • Publish Date: 21 May 2020

Share

How to cite

Statistics