Iterative Image Registration: Lucas & Kanade Revisited
Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges
Kentaro Toyama Vision Technology Group Microsoft Research (with additions by F. Devernay, INRIA)
History • • • • • • • • • • •
Image Registration
Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)
Applications • Stereo
Applications
Applications • Stereo • Dense optic flow
Applications • Stereo • Dense optic flow • Image mosaics
Applications • • • •
Stereo Dense optic flow Image mosaics Tracking
Applications • • • • •
Stereo Dense optic flow Image mosaics Tracking Recognition
L&K Derivation 1 Lucas & Kanade Derivation
=
+ →
−
L&K Derivation 1
+
≈
L&K Derivation 1
−
L&K Derivation 1
L&K Derivation 1
−
≈
−
≈
−
≈ ∈
L&K Derivation 1
L&K Derivation 1
−
≈ ∈
−
← ∈
L&K Derivation 1
←
+ ∈
L&K Derivation 1
− +
+
←
∈
L&K Derivation 1
+
←
+ ∈
+
− +
+
− +
+
L&K Derivation 1
− +
+
+
←
+ ∈
L&K Derivation 2 Lucas & Kanade Derivation
• Sum-of-squared-difference (SSD) error
Σ
!
Σ
!
∈R
≈
∈R
!
"
L&K Derivation 2 ∂ ∂
≈
Σ
≈
"
∈R
Σ
!
"
∈R
!
!
Σ
∈R
Comparison "
≈
≈
"
!
Σ
"
Σ Σ
"
!
Σ
"
Comparison ≈
≈
!
Σ
"
Generalizations
Σ Σ
"
Σ
! "
Original
Original • Dimension of image
Σ
∈R
$
!
0
Σ
∈R
$
!
0
1-dimensional
#
Generalization 1a • Dimension of image
Σ
• Dimension of image
!
$
∈R
2D:
Generalization 1b
0
=
Σ
!
$
∈R
=
Homogeneous 2D:
Problem A
0
Problem A Local minima:
Does the iteration converge?
Problem A Local minima:
Problem B Zero gradient:
≈
!Σ " ∈R
!
Σ
"
∈R
h is undefined if
Σ
∈R
"
is zero
%
Problem B Zero gradient:
Problem B’ Aperture problem:
≈
!Σ
∈R
∂ ∂
Σ
∈R
Problem B’ No gradient along one direction:
Solutions to A & B • Possible solutions: – Manual intervention – Zero motion default
! ∂ ∂
Solutions to A & B • Possible solutions: – Manual intervention
Solutions to A & B • Possible solutions: – Manual intervention – Zero motion default – Coefficient “dampening”
&
Solutions to A & B • Possible solutions:
Solutions to A & B • Possible solutions:
– Manual intervention – Zero motion default – Coefficient “dampening” – Reliance on good features
– Manual intervention – Zero motion default – Coefficient “dampening” – Reliance on good features – Temporal filtering
Solutions to A & B • Possible solutions:
Solutions to A & B • Possible solutions:
– Manual intervention – Zero motion default – Coefficient “dampening” – Reliance on good features – Temporal filtering – Spatial interpolation / hierarchical estimation
– Manual intervention – Zero motion default – Coefficient “dampening” – Reliance on good features – Temporal filtering – Spatial interpolation / hierarchical estimation – Higher-order terms
Original
Original • Transformations/warping of image
Σ
∈R
$
!
0
Σ
∈R
Translations:
$
! =
0
δ δ
'
Problem C
Generalization 2a • Transformations/warping of image
What about other types of motion?
Σ
∈R
Affine:
Generalization 2a
$
!
0
=
=
δ δ
Generalization 2b • Transformations/warping of image
Σ
!
∈R
)*+
=
=
δ δ
Planar perspective:
0
=
# &
Generalization 2b
%
'
Generalization 2c • Transformations/warping of image
Affine
+
Σ
∈R
Planar perspective:
=
# &
%
!
0
Other parametrized transformations
'
(
Generalization 2c
Generalization 2c
with
Other parametrized transformations
Problem B” ≈
!Σ " ∈R
Σ
∈R
Problem B” Generalized aperture problem:
! "
Generalized aperture problem:
,≈ !
!
!
Good features to track • Shi & Tomasi, 1994 • Avoid the generalized aperture problem: Good features maximize the smallest eigenvalue of JTJ • If f is limited to translation,
(Compare with Harris & Stephen)
Original
Σ
∈R
$
!
0
Original
Generalization 3
• Image type
Σ
∈R
• Image type
$
!
Σ
0
∈R
$
Grayscale images
Color images
Original
Original
!
0
!
0
• Constancy assumption
Σ
∈R
$
!
Σ
0
∈R
$
Brightness constancy
Problem C
Generalization 4a • Constancy assumption
What if illumination changes?
- α, β
Σ
∈R
$
!α
0
Linear brightness constancy
$β
Generalization 4a
Generalization 4b • Constancy assumption
λ
Σ
∈R
$
! λΤ
Illumination subspace constancy
Generalization 4b
Problem C’
• Constancy assumption
λ
Σ
∈R
$
! λΤ
What if the texture changes?
Illumination subspace constancy
Generalization 4c
Problem D
• Constancy assumption
λ
Σ
∈R
$
! λΤ
Texture subspace constancy
Convergence is slower as #parameters increases.
Solutions to D • Faster convergence:
Solutions to D • Faster convergence:
– Coarse-to-fine, filtering, interpolation, etc.
– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization
Solutions to D
Solutions to D
• Faster convergence: – Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation
• Faster convergence: – Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation • Difference decomposition
Solutions to D • Difference decomposition
Solutions to D • Difference decomposition
Solutions to D
Solutions to D
• Faster convergence:
• Faster convergence:
– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation • Difference decomposition
– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation • Difference decomposition
– Improvements in gradient descent
– Improvements in gradient descent • Multiple estimates of spatial derivatives
Solutions to D
Generalizations
• Multiple estimates / state-space sampling
Modifications made so far:
Σ
R
Original
$
!
Problem E
• Error norm
Σ
∈R
$
!
Squared difference:
0
What about outliers?
0
Generalization 5a
Original
• Error norm
Σρ
$
∈R
=
Robust error norm:
!
0
Σ
∈R
$
!
0
+
Original
Problem E’
• Image region / pixel weighting
Σ
∈R
$
!
0
. /0102341205 6734)8541 1 *7
Rectangular:
Generalization 6a
Problem E”
• Image region / pixel weighting
Σ
∈R
$
!
0
What about foreground occlusion?
77*6407+
#
Generalization 6b • Image region / pixel weighting
Σ
$
∈R
!
Generalizations Modifications made so far:
0
Σ
$
∈R
!
0
Weighted sum:
Generalization 6c • Image region / pixel weighting
Σ
$
∈R
!
Σ 0
Sampled:
Foresight • • • • • • • • • • •
Generalizations: Summary
Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)
$
∈R
Σρ ∈R
!
0
! λΒ
Summary • Generalizations
L&K ?
– Dimension of image – Image transformations / motion models – Pixel type – Constancy assumption – Error norm – Image mask
Y Y n Y n Y
%
Summary • Common problems: – Local minima – Aperture effect – Illumination changes – Convergence issues – Outliers and occlusions
Summary L&K ?
• Mitigation of aperture effect:
Y maybe Y Y n
– Manual intervention – Zero motion default – Coefficient “dampening” – Elimination of poor textures – Temporal filtering – Spatial interpolation / hierarchical – Higher-order terms
Summary • Better convergence: – Coarse-to-fine, filtering, etc. – Selective parametrization – Offline precomputation • Difference decomposition
– Improvements in gradient descent • Multiple estimates of spatial derivatives
L&K ? n n n n Y Y n
Hindsight L&K ? Y n maybe maybe maybe maybe
• • • • • • • • • • •
Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)
&
