608

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005

Variational Optical Flow Computation in Real Time Andrés Bruhn, Joachim Weickert, Christian Feddern, Timo Kohlberger, and Christoph Schnörr

Abstract—This paper investigates the usefulness of bidirectional multigrid methods for variational optical flow computations. Although these numerical schemes are among the fastest methods for solving equation systems, they are rarely applied in the field of computer vision. We demonstrate how to employ those numerical methods for the treatment of variational optical flow formulations and show that the efficiency of this approach even allows for real-time performance on standard PCs. As a representative for variational optic flow methods, we consider the recently introduced combined local-global method. It can be considered as a noise-robust generalization of the Horn and Schunck technique. We present a decoupled, as well as a coupled, version of the classical Gauß–Seidel solver, and we develop several multgrid implementations based on a discretization coarse grid approximation. In contrast, with standard bidirectional multigrid algorithms, we take advantage of intergrid transfer operators that allow for nondyadic grid hierarchies. As a consequence, no restrictions concerning the image size or the number of traversed levels have to be imposed. In the experimental section, we juxtapose the developed multigrid schemes and demonstrate their superior performance when compared to unidirectional multgrid methods and nonhierachical solvers. For the well-known 316 252 Yosemite sequence, we succeeded in computing the complete set of dense flow fields in three quarters of a second on a 3.06-GHz Pentium4 PC. This corresponds to a frame rate of 18 flow fields per second which outperforms the widely-used Gauß–Seidel method by almost three orders of magnitude. Index Terms—Computer vision, differential techniques, multigrid methods, optical flow, partial differential equations, structure tensor, variational methods.

I. INTRODUCTION A. Motivation

F

OR MORE THAN two decades, variational methods have played an important role in the area of optical flow estimation. Starting from the prototypical approach of Horn and Schunck [19], a rich variety of variational optical flow methods has been developed during recent years. A survey of these techniques ranging from linear homogenous to nonlinear anisotropic regularization is given in [31]. Apart from a convincing performance in terms of different error measures [2], [14], variational methods also offer several theoretical advantages: ExManuscript received September 2, 2003; revised June 8, 2004. This work was supported in part by the Deutsche Forschungsgemeinschaft (DFG) under the project SCHN 457/4-1. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Tamas Sziranyi. A. Bruhn, J. Weickert, and C. Feddern are with the Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, 66041 Saarbrücken, Germany (e-mail: [email protected]; [email protected]; [email protected]). T. Kohlberger and C. Schnörr are with the Computer Vision, Graphics and Pattern Recognition Group, Department of Mathematics and Computer Science, University of Mannheim, 68131 Mannheim, Germany (e-mail: [email protected]; [email protected]). Digital Object Identifier 10.1109/TIP.2005.846018

plicitly stating all model assumptions in a single energy functional, they allow for both a transparent modeling and a better understanding of the obtained results. Moreover, it is straightforward to design them in such a way that the continuous functional is rotationally invariant, and classical techniques from numerical analysis allow to approximate this rotation invariance also in the discrete algorithms with a high degree of accuracy. Using convex energy functionals also guarantees that the energy functional has a unique minimizer that can be found by globally convergent algorithms. Finally, the filling-in effect in variational methods yields dense flow field, whereas many other optical flow techniques require subsequent postprocessing steps to interpolate sparse flow estimates. However, these advantages are not for free. Variational methods require the minimization of an energy functional. This is often done by solving the linear or nonlinear system of equations that results from the discretization of the corresponding Euler–Lagrange equations. In order to solve these large sparse systems numerically, classical iterative methods such as the Gauß–Seidel algorithm are very popular [19]. While they are simple to implement, their convergence is not very fast, and often thousands of iterations may be necessary to get sufficiently close to the minimum of the energy functional. This is the reason why variational optical flow methods suffer from the reputation of being too slow for time-critical applications. The goal of the present paper is to show that it is possible to make variational optical flow methods suitable for real-time applications by combining them with state-of-the-art numerical techniques. As a demonstrator, we use the recently introduced combined local-global (CLG) method [10], [30], a variational technique that combines the advantages of two classical optical flow algorithms: the variational Horn and Schunck approach [19] and the local least-square technique of Lucas and Kanade [22]. For the CLG method, we derive fast numerical schemes based on so-called multigrid strategies [6], [7], [17], [29], [32]. Such techniques belong to the fastest numerical methods for solving linear systems of equations. We present our algorithms in detail and show that an appropriately designed full multigrid scheme leads to a speed up of almost three orders of magnitude compared to the widely used Gauß–Seidel methods. As a consequence, it becomes possible to compute 18 optical flow frames per second on a standard 3.06-GHz PC, when image sequences of size 316 252 pixels are used. Our paper is organized as follows. In Section II, we give a short review on the CLG model as a prototype for variational optical flow methods. Discretization aspects and suitable nonhierarchical solvers are discussed in Section III. Section IV explains in a detailed way how bidirectional multigrid schemes can be derived for the CLG method that allow the handling of nondyadic grid hierarchies. In Section V, these methods are jux-

1057-7149/$20.00 © 2005 IEEE

BRUHN et al.: VARIATIONAL OPTICAL FLOW COMPUTATION IN REAL TIME

taposed and their real-time capability is shown in a comparison to unidirectional multigrid schemes and nonhierarchical solvers. The paper is concluded with a summary in Section VI. B. Related Work It is quite common to use pyramid strategies for speeding up variational optical flow methods; see, e.g., [1], [23]. They use the solution at a coarse grid as initialization on the next finer grid. Such unidirectional techniques are also applied in order to escape from local minima in nonconvex energy functionals with multiple minima [24], [8]. These methods may be regarded as the simplest multigrid strategy, namely cascadic multigrid. The mathematical literature on multigrid methods shows that one can expect improved performance by using more advanced multigrid strategies that revisit coarse levels to obtain useful correction steps. Such so-called bidirectional techniques, however, are not used very frequently in the context of variational optical flow computation. First proposals go back to Glazer [16], Terzopoulos [28] and Enkelmann [12]. More recently, Ghosal and Vanˇek [15] developed an algebraic multigrid method for an anisotropic variational approach that can be related to Nagel’s method [25]. Zini et al. [34] proposed a conjugate gradient-based multigrid technique for an extension of the Horn and Schunck functional, while El Kalmoun and Rüde [20] presented a Galerkin approach [7] for the Horn and Schunck method itself. Furthermore, Borzi et al. [5] investigated a full approximation scheme (FAS) for a control formulation of a nonlinear optical flow problem. To the best of our knowledge, our paper is the first work that reports real-time performance for variational optical flow techniques on standard hardware. An earlier paper by the authors devoted to the same subject has been presented at a conference [9]. The present paper extends this work substantially in three aspects. 1) An additional relaxation scheme with significantly higher error reduction is introduced. 2) Nondyadic intergrid transfer operators allow for an application to image sequences of arbitrary size (not only powers of 2). 3) A more extensive experimental evaluation is carried out that comprises comparisons to both unidirectional multigrid schemes and to nonhierarchical solvers.

609

denotes the spatiotemporal gradient . The is the structure tensor [3], [13], [27] given by , where denotes convolution in each matrix component, and is a Gaussian with standard deviation . serves as regularization parameter. The weight is a presmoothed version of an origIn general, . The proposed method uses a inal image sequence Gaussian with standard deviation for this purpose. , the CLG approach comes down to the Horn and For it becomes the Lucas–Kanade Schunck method, and for algorithm. It combines the dense flow fields of Horn–Schunck with the high noise robustness of Lucas–Kanade. For a detailed evaluation, we refer to [10], [30], where it is shown that the CLG approach is fairly insensitive to noise and parameter variations and offers good performance in terms of low-average angular errors. For this paper, the CLG method only serves as a demonstrator for our numerical algorithms. In order to recover the optical flow field, the energy funchas to be minimized. As is well known from tional the calculus of variations [11], this can be done by solving its Euler–Lagrange equations and matrix

(2) (3) where denotes the Laplacian, and reflecting boundary conditions are applied, i.e., the normal derivative vanishes at image boundaries on

(4)

III. DISCRETIZATION Now, we are in a position to investigate typical algorithms for solving the Euler–Lagrange equations numerically. First, we present a finite difference discretization, and then we discuss two algorithms for solving the resulting linear systems of equations. A. Discrete Euler–Lagrange Equations

II. OPTICAL FLOW COMPUTATION WITH THE CLG APPROACH In [10] and [30], we have introduced the so-called combined local-global (CLG) method for optical flow computation. It combines the advantages of the global Horn and Schunck approach [19] and the local Lucas–Kanade method [22]. In order to describe the underlying idea behind the CLG method, let an image sequence, where denotes the location within a rectangular image domain and is the time. The CLG method computes the optical flow field at some time as the minimizer of the energy functional

where the vector field scribes the displacement,

is the spatial gradient

(1) de,

Let us now investigate a suitable discretization for the CLG method (2)–(3). To this end, we consider the unknown functions and on a rectangular pixel grid of cell size , and we denote by the approximation to at some . The main reason for the use of this pixel with general ordering index is the compactness of the resulting notation. Gaussian convolution is realized by discrete convolution with a truncated and renormalized Gaussian, where the truncation took place at three times the standard deviation. Symmetry and separability have been exploited in order to speed up these discrete convolutions. Spatial derivatives of the image data have been approximated using central differences, while temporal derivatives are approximated with a simple two-point the component of the stencil. Let us denote by in some pixel . Furthermore, let structure tensor denote the set of neighbors of pixel in the direction of axis .

610

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005

Then, a finite difference approximation to the Euler–Lagrange (2)–(3) with reflecting boundary conditions is given by

with

(11)

(5)

(6) for for the

and

. This constitutes a linear system of equations unknowns and .

B. Gauß–Seidel Method

(12)

The preceding linear system of equations has a sparse system matrix. It may be solved iteratively, e.g., by applying the Gauß–Seidel method [33]. Given a linear equation system and a splitting with diagonal matrix , lower triangular matrix and upper triangular matrix , then one iteration step of the Gauß–Seidel method reads (7) denotes the iteration index. Applied to the CLG where method, the corresponding iteration step is given by

One step of the pointwise coupled Gauß–Seidel method is computationally more expensive than the usual Gauß–Seidel method, since it requires to solve a 2 2 linear system in each pixel. In Section V, however, we will see that this additional effort does pay off. IV. EFFICIENT MULTIGRID ALGORITHM A. Motivation

(8)

(9) where

and . By , we denote the number of neighbors of pixel in direction of axis that belong to the image domain. C. Pointwise Coupled Gauß–Seidel Method Although and are computed sequentially at each pixel, this form of coupling between the two equations is rather weak. This may become particularly problematic for small , where and values of , when the data term are coupled, is dominating. As a remedy, one may think of simultaneously solving for both unknowns instead. Such a proceeding yields the so-called pointwise coupled Gauß–Seidel solver, a block Gauß–Seidel variant [18], where all unknowns denotes the at a pixel are updated synchronously. If at pixel , the pointwise coupled flow vector Gauß–Seidel method can be written as (10)

Common iterative solvers like the two presented Gauß–Seidel variants usually perform very well in removing the higher frequency parts of the error within the first iterations. This behavior is reflected in a good initial convergence rate. Because of their smoothing properties regarding the error, these solvers are referred to as smoothers. After some iterations, however, only low-frequency components of the error remain and the convergence slows down significantly. At this point, smoothers suffer from their local design and cannot attenuate efficiently low frequencies that have a sufficiently large wavelength in the spatial domain. Bidirectional multigrid methods [6], [7], [17], [29], [32] overcome this problem by creating a sophisticated fine-to-coarse hierarchy of equation systems with excellent error reduction properties. Low frequencies on the finest grid reappear as higher frequencies on coarser grids, where they can be removed successfully. This strategy allows such multigrid methods to compute accurate results much faster than nonhierarchical iterative solvers. Since we focus on the real-time computation of optical flow, we have developed several of these bidirectional multigrid algorithms for the CLG approach. B. Detailed Description of a Two-Grid Cycle Let us now explain our strategy in detail. We reformulate the linear system of equations given by (5)–(6) in matrix-vector notation as (13) where the index stands for the grid cell size notes the concatenated (column) vector is given by right-hand side

and and

de. The is a

BRUHN et al.: VARIATIONAL OPTICAL FLOW COMPUTATION IN REAL TIME

symmetric positive definite matrix. Let be the result computed by the chosen variant of the Gauß–Seidel smoother after iterations. Then, the error of the solution is given by (14) in order to correct the Evidently, one is interested in finding approximated solution . Since the error cannot be computed directly, we determine the residual error given by

611

and [20]. For a DCA, two different types of entries are distinguished. belonging to the discretized Laplacian Entries of depend on the grid cell size of the solution . Thus, the corresponding coarse grid entries are obtained by a simple adaptation of the grid size (20)

(15) instead. Since

is a linear operator, we have (16)

Solving this system of equations would give us the desired correction . Since high frequencies of the error have already been removed by our smoother, we can solve this system at a coarser level. For the sake of clarity, the notation for the coarser grid is chosen correspondingly to the original equation on the fine grid (13). Thus, the linear equation system (16) becomes

Having their origin in the structure tensor , all other enare independent of . Hence, their coarse grid tries of counterparts are obtained by a componentwise restriction of (21) After discussing the development of a suitable coarse grid representation, the equations system can be formulated in the following way:

(17) at the coarser level, where is the index for the new cell size with and . The right-hand side is a downsampled version of . At this point we have to make four decisions. 1) The cell size on the coarser grid has to be chosen. In contrast to the dyadic multgrid approaches in [16], [28], and to be a power [12], [34], and [9] that require of 2, our approach shall be applicable to any problem size. Therefore, we propose the following computation of . Let and be the number of cells on the fine grid in and direction. Then, the new cell size is given by

(22)

(23) for

, where

and . The corresponding ordinary and pointwise coupled Gauß–Seidel iteration steps can be derived directly from this system. on the coarse grid, a pro4) After solving has to be defined in order to longation operator back to the fine grid transfer the solution (24)

(18)

as prolonWe choose constant interpolation over (see Fig. 1). In particular, we have gation operator

and , where dewith . Thus, notes the smallest integer number with the total number of cells at the coarser grid is . This generalization to nondyadic grid hierarchies has also impact on the three remaining decisions. has to be defined that allows 2) A restriction operator the transfer of vectors from the fine to the coarse grid. By its application to the residual , we obtain the right hand side of the equation system on the coarser grid

which allows us to reuse the stored restriction weights for prolongation. The obtained correction can now be used for updating the approximated solution of the original equation on the fine grid

(19)

(26)

In our implementation, averaging over is used (see Fig. 1). for has to be created. To 3) A coarser version of the matrix this end, a so-called discretization coarse grid approximation (DCA) is performed [7]. Unlike [16], [28], [34], and [20], we keep strictly to its idea and rediscretize the coefficients of the Euler–Lagrange equations directly. As a result, we do not experience any convergence problems for an increasing number of levels as reported in [16]

iterations of the smoother are performed in order to Finally, remove high-error frequencies introduced by the prolongation . of

(25)

C. Advanced Multigrid Strategies In our explanation of the basic two-grid cycle, the question of an appropriate solver for the coarse grid system of equations was left open. A very elegant and efficient strategy is the use of a third grid that provides a correction for the second grid.

612

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005

Fig. 3. Example for a cascadic and full multigrid cycle for four levels. V cycles are denoted with v and refinement steps with c. TABLE I GAUß SEIDEL VERSUS POINTWISE-COUPLED GAUß SEIDEL Fig. 1. One-dimensional example for the restriction operator R with h=H = 3=4. prolongation operator P

and the

independent of the problem size. Thus, better convergence rates are obtained and the number of required cycles is reduced. Fig. 2. Example of V and W cycles for two, three, and four levels. Iterations on the original and residual equation are represented by large resp. small dots.

V. RESULTS

Such a hierarchical application of the presented two-grid cycle is called cycle. One may also think of performing a second, additional correction step per level. This yields the so-called cycle that has better convergence properties at the expense of slightly increased computational costs. Examples for and cycles are given in Fig. 2. In general, multiples of these cycles are required to reach the desired accuracy. Unidirectional multigrid methods—known as cascadic multigrid methods [4]—follow a totally different strategy. Instead of transferring the residual equations between the levels, they start with a coarse version of the original equation system. This system is then successively refined until the finest level is reached and the original problem is solved at full size (see Fig. 3). Thereby, coarse solutions serve as initial guesses for finer grids, so that fewer solver iterations are required at each refinement level. As a consequence, this strategy may provide a significant acceleration to nonhierarchical methods. Nevertheless, it does not allow to overcome the actual problem of iterative solvers: The slow attenuation of low-error frequencies. For this reason, it seems desirable to embed or cycles in the coarse-to-fine strategy of cascadic multigrid methods. Thus, bidirectional methods are obtained that benefit from good initial guesses and offer excellent error reduction properties at the same time. These so-called full multigrid methods [7] are known to deliver the best performance of all multigrid schemes. An example of such a strategy for four levels is shown in Fig. 3. Comparing the graphs of the full and the cascadic multigrid method, one can see that a cycle instead of a nonhierarchical relaxation scheme is used to solve the problem at each refinement level. This is the only but decisive difference between both strategies. One should note that in Fig. 3, the number of levels is only limited to four for better visualization. Since we take advantage of intergrid transfer operators that allow for nondyadic grid hierarchies, a full descent for all three strategies is always performed

Let us now evaluate the different multigrid strategies. To this end, all computations are carried out on a standard desktop PC with a 3.06-GHz Intel Pentium 4 CPU executing C/C++ code. In our first experiment, we compare the ordinary and the pointwise-coupled Gauß–Seidel method as part of a -cycle strategy with one presmoothing and one postsmoothing itera). In order to analyze the contion at each level ( vergence behavior of both numerical schemes with respect to parameter variations, test runs with four different parameter settings are performed. In each test run, we compute one flow field of the 512 512 Marble sequence by Otte and Nagel [26]. Its size allows the use of nine hierarchy levels down to a size of 2 2. The cycles are stopped when the relative error is below , where denotes the absolute error cycles and stands for the correct solution. after In Table I, the required number of cycles and the average are error reduction factor defined by listed. The superior performance of the pointwise coupled Gauß–Seidel method for decreasing values of is in accordance with the theoretical considerations from Section III. In those cases where the regularization is rather weak, one can also observe that a moderate use of the integration scale leads to a significant acceleration of the convergence. This smoothing process can allow a much more accurate coarse grid representation. Thus, the integration scale is not only useful to render the CLG approach more robust against noise, but also to improve its convergence within multigrid implementations. Since only the pointwise coupled Gauß–Seidel relaxation scheme is capable of handling cases of weak regularization properly, multigrid implementations based on the ordinary variant are not considered further in our evaluation. In our second experiment, different multigrid strategies for the pointwise-coupled Gauß–Seidel solver are compared. For this purpose, we perform test runs on two different image sequences. Besides the Marble sequence, the 200 200 pixels Of-

BRUHN et al.: VARIATIONAL OPTICAL FLOW COMPUTATION IN REAL TIME

613

TABLE II COMPARISON OF DIFFERENT MULTIGRID STRATEGIES

Fig. 4. (a) Left: Frame 16 of the Marble sequence. (b) Middle: Ground truth flow field between frame 16 and 17. (c) Right: Computed flow field by the CLG method ( = 2:6,  = 1:8, and  = 1000). Computing time with one W(1; 1) cycle: 165 ms.

fice sequence by Galvin et al. [14] is used. Designed for handling nondyadic grid hierarchies, our algorithm allows the traversal of eight levels for this sequence down to a problem size of 2 2. Table II shows the required number of cycles to reach , the average error reducthe desired precision of tion factor and the runtime for each implementation. cycles are denoted by , where and stand for the number of presmoothing and postsmoothing iterations. In the case of full multigrid strategies an additional number gives cycles are performed at each information on how many refinement level. The presented runtimes refer to the computation of one single flow field of the corresponding sequence. While strategies based on cycles require several iterations , only one or two cycles are to reach a relative error of sufficient to perform the same task. This is reflected in up to 50% shorter runtimes. The implemented full multigrid schemes cycles in those converges fastest, but cannot outperform the cases, where one of them is already sufficient to reach the desired accuracy. The corresponding flow fields for both sequences are presented in Figs. 4 and 5. Apart from an obviously good quality of the CLG model, one should note the fast computation times of 165 and 24 milliseconds per frame, respectively. These numbers give already an accurate impression of the capability of bidirectional multigrid methods regarding the acceleration of variational optical flow techniques. We would like to emphasize that visible deviations between the correct and the computed flow fields are due to limitations of the CLG model, not the multigrid . This is algorithm: The relative numerical error is below not giving any visible contribution. In our last experiment, we juxtapose representatives of three numerical categories: bidirectional multigrid strategies, unidirectional multigrid schemes, and nonhierarchical iterative solvers. Based on the evaluation in Table II, we pick the full cycle to represent multigrid implementation with one the class of bidirectional multigrid methods. As nonhierarchical solvers, we consider the ordinary Gauß–Seidel method as well as its popular successive overrelaxation (SOR) variant [33] that accelerates the original method by a weighted extrapolation of its results. Moreover, cascadic multigrid implementations based on the preceding solvers participate for the class of unidirectional multigrid methods. For this last experiment, we use the famous Yosemite sequence without clouds that was originally created by Lynn Quam. While its image size of 316 252 raises problems for dyadic multigrid methods, our generalized

Fig. 5. (a) Left: Frame 10 of the Office sequence. (b) Middle: Ground truth flow field between frame 10 and 11. (c) Right: Computed flow field by the CLG method ( = 0:72,  = 1:8, and  = 2700). Computing time with one FMG(1; V(2; 2)) cycle: 24 ms. TABLE III PERFORMANCE BENCHMARK

prolongation and restriction operators allows a full descend of nine hierarchy levels down to a problem size of 2 1. Table III shows the results for all five implementations. The presented runtimes refer to the complete set of 14 flow fields of the Yosemite sequence. As one can see, we are able to compute the optical flow for sequences of 316 252 pixels in real time. In order to achieve the measured rate of 18 frames per second, almost three million unknowns have to be calculated every second. With the speedup of 708, our bidirectional multigrid scheme is not only three orders of magnitude faster than a standard implementation based on the widely-used Gauß–Seidel method, but also one order of magnitude more efficient than SOR. The table shows also that unidirectional multigrid schemes provide a significant acceleration to their basic solvers, but are outperformed by bidirectional ones. In terms of iterations, the advantage of our full multigrid implementation over nonhierarchical solvers is even more drastical: While 21 931 Gauß–Seidel iterations are required to reach the desired accuracy, one single full multigrid cycle is sufficient. Qualitative results for this comparison are presented in Fig. 6 where the computed flow field for the Yosemite sequence is shown. We observe that the CLG method matches the ground truth very well. Again, one should keep in mind that the computation of such a single flow field of size 316 252 takes only

614

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005

Our ongoing work on numerical methods for variational optical flow computation focuses on further performance enhancements by parallelization techniques [21] and on the investigation of highly efficient schemes for nonlinear methods. ACKNOWLEDGMENT The authors would like to thank Prof. U. Rüde and Dr. M. El Kalmoun from the University of Erlangen for interesting multigrid discussions. They would also like to thank Prof. B. Finkbeiner from Saarland University for providing the testing environment. REFERENCES

Fig. 6. (a) Top left: Frame 8 of the Yosemite sequence without clouds. (b) Top right: Ground truth flow field between frame 8 and 9. (c) Bottom left: Computed flow field by the CLG method ( = 1:5,  = 1:8, and  = 800). Computing time with one FMG(1; V(2; 2)) cycle: 54 ms per frame. (d) Bottom right: Error map of the computed flow field. The average angular error is 2.63 degrees.

54 ms. This short time is sufficient to compute a dense flow field with a competitive average angular error of 2.63 . VI. SUMMARY AND CONCLUSIONS We have demonstrated that bidirectional multigrid methods are capable of accelerating variational optical flow techniques to such an extent that they allow for real-time performance on standard hardware. To this end, we have developed several multigrid implementations for the CLG method based on a discretization coarse grid approximation (DCA). Using nondyadic intergrid transfer operators, we were able to set up complete grid hierarchies for arbitrary image sizes, not only powers of 2. This improves the practical applicability and efficiency of the presented approach in a substantial way. We have also investigated the use of decoupled and coupled versions of the classical Gauß–Seidel method as basic iterative solvers for our multigrid schemes. By juxtaposing implementations for both versions, we have shown that only the coupled variant is able to handle cases of weak regularization efficiently. Comparisons between different bidirectional multigrid strategies allowed us to identify the fastest numerical scheme for our purpose: full multigrid. In a final experiment, the superiority of this implementation became obvious. By computing 18 flow fields of size 316 252 within a single second, it outperformed unidirectional multigrid methods as well as nonhierarchical relaxation schemes by up to three orders of magnitude. These results have shown that variational methods for optical flow computation do not only belong to the qualitatively best strategies for motion estimation, they are also highly efficient for computing dense flow fields in real time. A single iteration of a carefully designed full-multigrid cycle reduces the numerical error to such a small amount that it is negligible in comparison to all errors that are introduced by inaccurate model assumptions.

[1] P. Anandan, “A computational framework and an algorithm for the measurement of visual motion,” Int. J. Comput. Vis., vol. 2, pp. 283–310, 1989. [2] J. L. Barron, D. J. Fleet, and S. S. Beauchemin, “Performance of optical flow techniques,” Int. J. Comput. Vis., vol. 12, no. 1, pp. 43–77, Feb. 1994. [3] J. Bigün, G. H. Granlund, and J. Wiklund, “Multidimensional orientation estimation with applications to texture analysis and optical flow,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 8, pp. 775–790, Aug. 1991. [4] F. Bornemann and P. Deuflhard, “The cascadic multigrid method for elliptic problems,” Numer. Math., vol. 75, pp. 135–152, 1996. [5] A. Borzi, K. Ito, and K. Kunisch, “Optimal control formulation for determining optical flow,” SIAM J. Sci. Comput., vol. 24, no. 3, pp. 818–847, 2002. [6] A. Brandt, “Multi-level adaptive solutions to boundary-value problems,” Math. Comput., vol. 31, no. 138, pp. 333–390, Apr. 1977. [7] W. L. Briggs, V. E. Henson, and S. F. McCormick, A Multigrid Tutorial, second ed. Philadelphia, PA: SIAM, 2000. [8] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “Highy accuracy optical flow estimation based on a theory for warping,” in Proc. Computer Vision ECCV, vol. 3024, Springer LCNS, May 2004, pp. 25–26. [9] A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, and C. Schnörr, “Realtime optic flow computation with variational methods,” in Proc. Computer Analysis of Images and Patterns, vol. 2756, Springer LCNS, Aug. 2003, pp. 222–229. [10] A. Bruhn, J. Weickert, and C. Schnörr, “Combining the advantages of local and global optic flow methods,” in Proc. Pattern Recognition, Springer LCNS, Sep. 2002, pp. 454–462. [11] R. Courant and D. Hilbert, Methods of Mathematical Physics. New York: Interscience, 1953, vol. 1. [12] W. Enkelmann, “Investigation of multigrid algorithms for the estimation of optical flow fields in image sequences,” Comput. Vis., Graph., Image Process., vol. 43, pp. 150–177, 1987. [13] W. Förstner and E. Gülch, “A fast operator for detection and precise location of distinct points, corners and centres of circular features,” in Proc. ISPRS Intercommission Conf. Fast Processing of Photogrammetric Data, Interlaken, Switzerland, Jun. 1987, pp. 281–305. [14] B. Galvin, B. McCane, K. Novins, D. Mason, and S. Mills, “Recovering motion fields: An analysis of eight optical flow algorithms,” presented at the Brit. Machine Vision Conf., Southampton, U.K., Sep. 1998. ˇ Vanˇek, “Scalable algorithm for discontinuous optical [15] S. Ghosal and P. C. flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 2, pp. 181–194, Feb. 1996. [16] F. Glazer, “Multilevel relaxation in low-level computer vision,” in Multiresolution Image Processing and Analysis, A. Rosenfeld, Ed. Berlin, Germany: Springer, 1984, pp. 312–330. [17] W. Hackbusch, Multigrid Methods and Applications. New York: Springer, 1985. [18] , Applied Mathematical Sciences. New York: Springer, 1993, vol. 95. [19] B. Horn and B. Schunck, “Determining optical flow,” Artif. Intell., vol. 17, pp. 185–203, 1981. [20] M. Kalmoun and U. Rüde, “A variational multigrid for computing the optical flow,” in Vision, Modeling and Visualization, T. Ertl, B. Girod, G. Greiner, H. Niemann, H.-P. Seidel, E. Steinbach, and R. Westermann, Eds. Geneva, Switzerland: IOS, 2003, pp. 577–584.

BRUHN et al.: VARIATIONAL OPTICAL FLOW COMPUTATION IN REAL TIME

[21] T. Kohlberger, C. Schnörr, A. Bruhn, and J. Weickert, “Parallel variational motion estimation by domain decomposition and cluster computing,” in Proc. Computer Vision ECCV, Springer LCNS, May 2004, pp. 205–216. [22] B. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. 7th Int. Joint Conf. Artificial Intelligence, Vancouver, BC, Canada, Aug. 1981, pp. 674–679. [23] M. R. Luettgen, W. C. Karl, and A. S. Willsky, “Efficient multiscale regularization with applications to the computation of optical flow,” IEEE Trans. Image Process., vol. 3, no. 1, pp. 41–64, Feb. 1994. [24] E. Mémin and P. Pérez, “Dense estimation and object-based segmentation of the optical flow with robust techniques,” IEEE Trans. Image Process., vol. 7, no. 5, pp. 703–719, May 1998. [25] H.-H. Nagel, “Constraints for the estimation of displacement vector fields from image sequences,” in Proc. 8th Int. Joint Conf. Artificial Intelligence, vol. 2, Karlsruhe, Germany, Aug. 1983, pp. 945–951. [26] M. Otte and H.-H. Nagel, “Estimation of optical flow based on higherorder spatiotemporal derivatives in interlaced and noninterlaced image sequences,” Artif. Intell., vol. 78, pp. 5–43, 1995. [27] A. R. Rao and B. G. Schunck, “Computing oriented texture fields,” CVGIP: Graph. Models Image Process., vol. 53, pp. 157–185, 1991. [28] D. Terzopoulos, “Image analysis using multigrid relaxation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. PAMI-8, no. 2, pp. 129–139, Mar. 1986. [29] U. Trottenberg, C. Oosterlee, and A. Schüller, Multigrid. San Diego, CA: Academic, 2001. [30] J. Weickert, A. Bruhn, and C. Schnörr, “Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods,” Int. J. Comput. Vis., vol. 61, no. 3, pp. 211–231, Feb./Mar. 2005. [31] J. Weickert and C. Schnörr, “A theoretical framework for convex regularizers in PDE-based computation of image motion,” Int. J. Comput. Vis., vol. 45, no. 3, pp. 245–264, Dec. 2001. [32] P. Wesseling, An Introduction to Multigrid Methods. Chichester, U.K.: Wiley, 1992. [33] D. M. Young, Iterative Solution of Large Linear Systems. New York: Academic, 1971. [34] G. Zini, A. Sarti, and C. Lamberti, “Application of continuum theory and multi-grid methods to motion evaluation from 3D echocardiography,” IEEE Trans. Ultrason., Ferroelect., Freq. Control, vol. 44, no. 3, pp. 297–308, Mar. 1997.

Andrés Bruhn received the M.Sc. degree in computer engineering from the University of Mannheim, Mannheim, Germany in 2001. He is currently pursuing the Ph.D. degree in computer science at Saarland University, Saarbrücken, Germany. He was a Research Assistant at the University of Mannheim from October to November 2001. Since December 2001, he has been with the Mathematical Image Analysis Group, Department of Mathematics and Computer Science, Saarland University. His current research interests include optical flow estimation, diffusion filtering, and fast numerical schemes for image processing and computer vision. Mr. Bruhn won the prestigious Longuet–Higgins Prize at the European Conference on Computer Vision in 2004.

615

Joachim Weickert received the M.Sc. and Ph.D. degrees in mathematics from the University of Kaiserslautern, Kaiserslautern, Germany, and the habilitation degree in computer science from the University of Mannheim, Mannheim, Germany, in 1991, 1996, and 2001, respectively. He was a Research Assistant at the University of Kaiserslautern; a Postdoctoral Researcher at the University of Utrecht, Utrecht, The Netherlands, and the University of Copenhagen, Copenhagen, Denmark; and an Assistant Professor at the University of Mannheim. Currently, he is a Full Professor of mathematics and computer science at Saarland University, Saarbrücken, Germany. His research interests include image processing, computer vision, partial differential equations, and scientific computing.

Christian Feddern received the M.Sc. degree in computer science from Saarland University, Saarbrücken, Germany, in 2000, where he is currently pursuing the Ph.D. degree in computer science. From 1998 to 2001, he was an IT Manager for Orbis and became Head of the IT Department for eXirius in 2001. Since 2002, he has been a Research Assistant with the Mathematical Image Analysis Group, Department of Mathematics and Computer Science, Saarland University. His current research focuses on methods for tensor valued image processing.

Timo Kohlberger studied computer engineering at the University of Mannheim, Mannheim, Germany, where, during his studies, he was supported by the German National Merit Foundation, and from where he received the Diplom (Master) degree in computer science in 2001. Since October 2001, he has been with the CVGPR Group of the University of Mannheim. His present research interests include the design of novel algorithms and parallelization techniques applied to image-motion computation and computer vision.

Christoph Schnörr received the Dipl.–Ing. degree in electrical engineering and the Dr.rer.nat. degree in computer science from the Technical University of Karlsruhe, Karlsruhe, Germany, and the Habilitation degree in computer science from the University of Hamburg, Hamburg, Germany, in 1987, 1991, and 1998, respectively. He held the position of Researcher at the Fraunhofer Institute for Information and Data Processing (IITB), Karlsruhe, from 1987 to 1992 and Researcher and Assistant Professor within the Cognitive Systems Group, University of Hamburg. During 1996, he was a Visiting Researcher at the Computer Vision and Active Perception Laboratory, KTH, Stockholm, Sweden. Since 1999, he has been with the Department of Mathematics and Computer Science, University of Mannheim, Mannheim, Germany, where he set up and is the Head of the Computer Vision, Graphics, and Pattern Recognition Group. He serves on the editorial board of the Journal of Mathematical Imaging and Vision. His research interests include computer vision, pattern recognition, and related aspects of mathematical modeling and optimization. Dr. Schnörr is a member of DAGM, SIAM, and SIAGIS.