Iterative Image Registration: Lucas & Kanade Revisited
Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges
Kentaro Toyama Vision Technology Group Microsoft Research
History • • • • • • • • • • • LK
Image Registration
Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
Applications • Stereo
Applications
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
1
Applications
Applications
• Stereo • Dense optic flow
LK
BAHH
ST
SC
• Stereo • Dense optic flow • Image mosaics
S
BJ
HB
BL
G
SI
CET
LK
BAHH
ST
Applications • • • •
LK
ST
SC
• • • • •
S
BJ
HB
S
BJ
HB
BL
G
SI
CET
BL
G
SI
CET
Applications
Stereo Dense optic flow Image mosaics Tracking
BAHH
SC
BL
G
SI
CET
LK
Stereo Dense optic flow Image mosaics Tracking Recognition
BAHH
ST
SC
?
S
BJ
HB
L&K Derivation 1 Lucas & Kanade Derivation
I0(x)
#1
2
L&K Derivation 1 h
L&K Derivation 1 h
I0(x)
I0(x+h)
I(x)
L&K Derivation 1 h
L&K Derivation 1 I0(x)
I(x)
I0(x)
I(x)
L&K Derivation 1
R
L&K Derivation 1 h0
I0(x)
I(x)
I0(x)
I0(x)
I(x)
3
L&K Derivation 1
L&K Derivation 1
I0(x+h0)
I(x)
I0(x+h1)
I(x)
L&K Derivation 1
L&K Derivation 1
I0(x+hk)
I(x)
I0(x+hf)
I(x)
L&K Derivation 2 Lucas & Kanade Derivation
• Sum-of-squared-difference (SSD) error
E(h) = Σ [ I(x) - I0(x+h) ]2 xεR
#2
E(h)
Σ [ I(x) - I0(x) - hI0’(x) ]2 xεR
4
L&K Derivation 2
Comparison
Σ 2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2]
Σ
x
h
xεR
w(x)[I(x) - I0(x)] I0’(x)
Σ
=0 h
Σ
xεR
I0’(x)(I(x) - I0(x))
Σ
xεR
Σ
x
h
I0’(x)2
x
w(x)
I0’(x)[I(x) - I0(x)]
Σ
x
I0’(x)2
Comparison h
h
Σ
x
w(x)[I(x) - I0(x)] I0’(x)
Σ Σ
x
x
Generalizations
w(x)
I0’(x)[I(x) - I0(x)]
Σ
x
I0’(x)2
Original
Original • Dimension of image
E( h ) =
Σ [ I( x + h ) -
x εR
I0 (x) ] 2
E( h ) =
Σ [ I( x + h ) -
x εR
I0 (x) ] 2
1-dimensional
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
5
Generalization 1a
Generalization 1b
• Dimension of image
E( h ) =
• Dimension of image
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
E( h ) =
2D:
LK
BAHH
ST
Σ [ I( x + h ) -
I0 (x) ] 2
xεR
Homogeneous 2D:
SC
S
BJ
HB
BL
G
SI
CET
LK
BAHH
ST
SC
Problem A
S
BJ
HB
BL
G
SI
CET
Problem A Local minima:
Does the iteration converge?
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
Problem A
Problem B Zero gradient:
Local minima:
-Σ I0’(x)(I(x) - I0(x)) xεR
h
Σ
xεR
h is undefined if
I0’(x)2
Σ
xεR LK
BAHH
ST
SC
S
BJ
I0’(x)2 HB
BL
is zero G
SI
CET
6
Problem B
Problem B’
Zero gradient:
Aperture problem:
hy
-Σ
(x)(I(x) - I0(x))
xεR
2
Σ
xεR
? LK
BAHH
Problem B’
ST
BAHH
Solutions to A & B
BL
G
SI
CET
SC
S
BJ
HB
BL
G
SI
CET
G
SI
CET
• Possible solutions:
– Manual intervention – Zero motion default
S
ST
Solutions to A & B
• Possible solutions:
SC
HB
– Manual intervention
LK
ST
BJ
• Possible solutions:
?
BAHH
S
Solutions to A & B
No gradient along one direction:
LK
SC
– Manual intervention – Zero motion default – Coefficient “dampening”
BJ
HB
BL
G
SI
CET
LK
BAHH
ST
SC
S
BJ
HB
BL
7
Solutions to A & B
Solutions to A & B
• Possible solutions: – – – –
LK
• Possible solutions:
Manual intervention Zero motion default Coefficient “dampening” Reliance on good features
BAHH
ST
SC
S
BJ
– – – – –
HB
BL
G
SI
CET
LK
Manual intervention Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering
BAHH
Solutions to A & B
LK
ST
SC
S
S
BJ
HB
BL
G
SI
CET
• Possible solutions:
Manual intervention Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation
BAHH
SC
Solutions to A & B
• Possible solutions: – – – – – –
ST
BJ
HB
BL
G
SI
– – – – – – – CET
LK
Manual intervention Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation Higher-order terms
BAHH
ST
SC
Original
S
BJ
HB
BL
G
SI
CET
Original • Transformations/warping of image
E( h ) =
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
E( h ) =
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
Translations:
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
8
Problem C
Generalization 2a • Transformations/warping of image
What about other types of motion?
E(A, h) =
Σ [ I(Ax+ h ) -
xεR
I0 (x) ] 2
Affine:
LK
BAHH
ST
Generalization 2a
SC
S
BJ
HB
BL
G
SI
CET
Generalization 2b • Transformations/warping of image
E( A) = Affine:
Σ [ I(
xεR
A x ) - I0 (x) ] 2
Planar perspective:
LK
BAHH
ST
Generalization 2b
SC
S
BJ
HB
BL
G
SI
CET
Generalization 2c • Transformations/warping of image
Affine
+
E( h ) = Planar perspective:
Σ [ I( f(x, h)) -
xεR
I0 (x) ] 2
Other parametrized transformations
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
9
Generalization 2c
Problem B” -Σ I0’(x)(I(x) - I0(x)) xεR
h
Σ
xεR
I0’(x)2
Generalized aperture problem:
~ h
Other parametrized transformations
LK
BAHH
-(JTJ)-1 J (I(f(x,h)) - I0(x)) SC
ST
Problem B”
S
BJ
HB
BL
G
SI
CET
Original
Generalized aperture problem:
E( h ) =
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
?
Original
Generalization 3
• Image type
E( h ) =
• Image type
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
E( h ) =
Σ ||I( x + h ) -
xεR
Grayscale images
LK
BAHH
ST
SC
S
BJ
HB
I0(x) ||2
Color images
BL
G
SI
CET
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
10
Original
Original • Constancy assumption
E( h ) =
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
E( h ) =
Σ [ I( x + h ) -
I0 (x) ] 2
xεR
Brightness constancy
LK
BAHH
ST
Problem C
SC
S
BJ
HB
BL
G
SI
CET
Generalization 4a • Constancy assumption
What if illumination changes?
E( h,α,β )= Σ [ I( x + h ) - αI0(x)+β] 2 xεR
Linear brightness constancy
LK
BAHH
ST
Generalization 4a
SC
S
BJ
HB
BL
G
SI
CET
Generalization 4b • Constancy assumption
E ( h,λ) =
Σ [ I( x + h ) - λΤB(x)] 2
xεR
Illumination subspace constancy
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
11
Problem C’
Generalization 4c • Constancy assumption
E ( h,λ) =
What if the texture changes?
Σ [ I( x + h ) - λΤB(x)] 2
xεR
Texture subspace constancy
LK
BAHH
ST
Problem D
SC
S
BJ
HB
BL
G
SI
CET
SI
CET
SI
CET
Solutions to D • Faster convergence: – Coarse-to-fine, filtering, interpolation, etc.
Convergence is slower as #parameters increases.
LK
BAHH
ST
Solutions to D
ST
SC
S
BJ
HB
BL
G
• Faster convergence:
– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization
BAHH
S
Solutions to D
• Faster convergence:
LK
SC
BJ
HB
BL
G
– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation
SI
CET
LK
BAHH
ST
SC
S
BJ
HB
BL
G
12
Solutions to D
Solutions to D
• Faster convergence:
• Difference decomposition
– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation • Difference decomposition
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
Solutions to D
Solutions to D
• Difference decomposition
• Faster convergence: – Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation • Difference decomposition
– Improvements in gradient descent
LK
Solutions to D
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
Solutions to D
• Faster convergence:
• Multiple estimates / state-space sampling
– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation • Difference decomposition
– Improvements in gradient descent • Multiple estimates of spatial derivatives
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
13
Generalizations
Original • Error norm
Modifications made so far:
Σ [ I( x + h ) -
x εR
I0 (x) ] 2
E( h ) =
Σ [ I( x + h ) -
I0 (x) ] 2
xεR
Squared difference:
LK
BAHH
ST
Problem E
SC
S
BJ
HB
BL
G
SI
CET
SI
CET
Generalization 5a • Error norm
E( h ) =
What about outliers?
Σ ρ ( I( x + h ) -
xεR
I0 (x) )
Robust error norm:
LK
BAHH
ST
SC
Original
S
BJ
HB
BL
G
Original • Image region / pixel weighting
E( h ) =
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
E( h ) =
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
Rectangular:
LK
BAHH
ST
SC
S
BJ
HB
BL
G
SI
CET
14
Problem E’
Generalization 6a • Image region / pixel weighting
E( h ) =
What about background clutter?
Σ [ I( x + h ) -
xεR
I0 (x) ] 2
Irregular:
LK
BAHH
ST
Problem E”
SC
S
BJ
HB
BL
G
SI
CET
Generalization 6b • Image region / pixel weighting
E( h ) =
What about foreground occlusion?
Σ [ I( x + h ) -
xεR
I0 (x) ] 2w(x)
Weighted sum:
LK
Generalizations
BAHH
ST
Σ [ I( x + h ) -
S
BJ
HB
BL
G
SI
CET
Generalizations: Summary
Modifications made so far:
x εR
SC
I0 (x) ]
2
E( h ) =
E( h ) =
Σ [ I( x + h ) -
x εR
I0 (x) ] 2
Σ ρ ( I( f(x, h)) - λΒ(x) ) w(x)
xεR
15
Foresight • • • • • • • • • • • LK
Summary
Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) BAHH
ST
SC
S
BJ
HB
BL
• Generalizations – – – – – – G
SI
– – – – –
Local minima Aperture effect Illumination changes Convergence issues Outliers and occlusions
Summary L&K ?
• Mitigation of aperture effect:
Y maybe Y Y n
– – – – – – –
Summary • Better convergence: – Coarse-to-fine, filtering, etc. – Selective parametrization – Offline precomputation • Difference decomposition
– Improvements in gradient descent • Multiple estimates of spatial derivatives
Y Y n Y n Y
CET
Summary • Common problems:
Dimension of image Image transformations / motion models Pixel type Constancy assumption Error norm Image mask
L&K ?
Manual intervention Zero motion default Coefficient “dampening” Elimination of poor textures Temporal filtering Spatial interpolation / hierarchical Higher-order terms
L&K ? n n n n Y Y n
Hindsight L&K ? Y n maybe maybe maybe maybe
• • • • • • • • • • •
Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)
16
