TU Berlin

Communication Systems GroupRobust Local Optical Flow

Page Content

to Navigation

Robust Local Optical Flow


The Robust Local Optical Flow (RLOF) is a sparse optical flow and feature tracking method. Our research is motivated by the problem of local motion estimation via robust regression with linear models. The main objective is to provide real-time capability, accurate and scalable motion estimation solution. The main advantage of the RLOF approach is the adjustable runtime and computational complexity which is in contrast to most common optical flow methods linearly dependend on the number of motion vectors (features) to be estimated. The RLOF implementation can be seen as an improved pyramidal iterative Lucas-Kanade and includes a set of improving modules. The main improvements in respect to the pyramidal iterative Lucas-Kanade are:

  • A more robust redecending M-estimator framework to improve the accuracy at motion boundaries and appearing and disappearing pixels.
  • An adaptive support region strategies to improve the accuracy at motion boundaries to reduce the corona effect, i.e oversmoothing of the PLK at motion/object boundaries. The cross-based segementation strategy uses a simple segmenation approach to obtain the optimal shape of the support region.
  • To deal with illumination changes (outdoor sequences and shadow) the intensity constancy assumption based optical flow equation has been adopt with the Gennert and Negahdaripour illumination model.
  • By using a global motion prior initialization of the iterative refinement the accuracy could be significantly improved for large displacements. 
  • The application of a sparse-to-dense interpolation scheme allows to compute dense optical flow fields in real-time.

Code: https://github.com/tsenst/RLOFLib

RLOF is now part of OpenCV Contribution GITdocs

Dissertation: Tobias Senst, "Estimation and Analysis of Motion in Video Data", 17.06.2019



Robust Local Optical Flow: Dense Motion Vector Field Interpolation


Optical flow methods integrating sparse point correspondences have made significant contribution in the field of optical flow estimation. Especially for the goal of estimating motion accurately and efficiently, sparse-to-dense interpolation schemes for feature point matches have shown outstanding performances. Concurrently, local optical flow methods have been significantly improved with respect to long-range motion estimation in environments with varying illumination. This motivates us to propose a sparse-to-dense approach based on the Robust Local Optical Flow method.


Geister,J.,Senst,T.,Sikora,T. Robust Local Optical Flow: Dense Motion Vector Field Interpolation. PCS, 2016

Robust local optical flow: Long-range motions and varying illuminations


Sparse motion estimation with local optical flow methods is fundamental for a wide range of computer vision application. Classical approaches like the pyramidal Lucas-Kanade method (PLK) or more sophisticated approaches like the Robust Local Optical Flow (RLOF) fail when it comes to environments with illumination changes and/or long-range motions. In this work we focus on these limitations and propose a novel local optical flow framework taking into account an illumination model to deal with varying illumination and a prediction step based on a perspective global motion model to deal with long-range motions. 


Senst,T.,Geister,J.,Sikora,T. Robust local optical flow: Long-range motions and varying illuminations. ICIP, 2016

Cross based Robust Local Optical Flow


With this work we propose an extension to these methods that improves the accuracy especially at object boundaries. This extension makes use of the cross based variable support region generation accounting for local intensity discontinuities


Senst,T., Borgmann, T., Keller, I., and Sikora,T. Cross based Robust Local Optical Flow. ICIP, 2014

Robust Local Optical Flow Estimation using Bilinear Equations for Sparse Motion Estimation


This work presents a theoretical framework to decrease the computation effort of the Robust Local Optical Flow method which is based on the Lucas Kanade method. We show mathematically, how to transform the iterative scheme of the feature tracker into a system of bilinear equations and thus estimate the motion vectors directly by analyzing its zeros. Furthermore, we show that it is possible to parallelise our approach efficiently on a GPU, thus, outperforming the current OpenCV-OpenCL implementation of the pyramidal Lucas Kanade method in terms of runtime and accuracy.


Senst, T.,Geister,J., Keller, I., and Sikora,T. Robust Local Optical Flow Estimation using Bilinear Equations for Sparse Motion Estimation. ICIP, 2013

Clustering Motion for Real-Time Optical Flow based Tracking


The selection of regions or sets of points to track is a key task in motion-based video analysis, which has significant performance effects in terms of accuracy and computational efficiency. Computational efficiency is an unavoidable requirement in video surveillance applications. Well established methods, e.g. Good Features to Track, select points to be tracked based on appearance features such as cornerness and therefore neglecting the motion exhibited by the selected points. In this paper, we propose an interest point selection method that takes into account the motion of previously tracked points in order to constrain the number of point trajectories needed. By defining pair-wise temporal affinities between trajectories and representing them in a minimum spanning tree, we achieve a very efficient clustering. The number of trajectories assigned to each motion cluster is adapted by initializing and removing tracked points by means of feed-back.


Senst, T., Heras Evangelio, R., Keller, I., and Sikora,T. Clustering Motion for Real-Time Optical Flow based Tracking. AVSS, 2012

Performance Evaluation of Feature Detection for Local Optical Flow Tracking

Due to its high computational efficiency the Kanade Lucas Tomasi feature tracker is still widely accepted and a utilized method to compute sparse motion fields or trajectories in video sequences. This method is made up of a Good Feature To Track feature detection and a pyramidal Lucas Kanade feature tracking algorithm. It is well known that the Good Feature To Track takes into account the Aperture Problem, but it does not consider the Generalized Aperture Problem. In this paper we want to provide an evaluation of a set of alternative feature detection methods. These methods are taken from feature matching techniques like FAST, SIFT and MSER.


Senst, T.,Unger,B., Keller, I., and Sikora,T. Performance Evaluation of Feature Detection for Local Optical Flow Tracking, ICPRAM, 2012

Efficient Real-Time Local Optical Flow Estimation by Means of Integral Projections


In this paper we present an approach for the efficient computation of optical flow fields in real-time and provide implementation details. Proposing a modification of the popular Lucas-Kanade energy functional based on integral projections allows us to speed up the method notably. We show the potential of this method which can compute dense flow fields of 640x480 pixels at a speed of 4 fps in a GPU implementation based on the OpenCL framework. Working on sparse optical flow fields of up to 17,000 points, we reach execution times of 70 fps.


Senst, T., Eiselein, V., P├Ątzold, M., and Sikora,T. Efficient Real-Time Local Optical Flow Estimation by Means of Integral Projections, ICIP, 2011

II-LK-A Real-Time Implementation for sparse Optical Flow


In this work we present an approach to speed up the computation of sparse optical flow fields by means of integral images and provide implementation details. Proposing a modification of the Lucas-Kanade energy functional allows us to use integral images and thus to speed up the method notably while affecting only slightly the quality of the computed optical flow. The approach is combined with an efficient scanline algorithm to reduce the computation of integral images to those areas where there are features to be tracked. The proposed method can speed up current surveillance algorithms used for scene description and crowd analysis.


Senst, T., Eiselein, V., and Sikora, T. II-LK-A Real-Time Implementation for sparse Optical Flow, ICIAR 2010


Quick Access

Schnellnavigation zur Seite über Nummerneingabe