Introduction to Computer Vision
Content
1.State of the art 2.Why Computer Vision is Challenging ? 3.Related fields 4.Typical tasks of computer vision 5.Computer vision systems 6.Further reading
2014
T. Chateau
T. Chateau
Institut Pascal
1
2
Computer Vision: Definition
Computer Vision: State of the art
Computer vision is a field that includes methods for acquiring, processing, analyzing, and understanding images and, in general, highdimensional data from the real world in order to produce numerical or symbolic information, e.g., in the forms of decisions
1950: Image processing 1960: first image processing algorithms
Processing
Acquiring
High Dimensional data
binary and edge based technics Analyzing
Understanding T. Chateau
Institut Pascal
3
T. Chateau
Institut Pascal
4
Computer Vision: State of the art
Computer Vision: State of the art
1990: projective geometry, a major contribution to Computer Vision
T. Chateau
>2000: toward robust and realtime algorithms
Institut Pascal
T. Chateau
5
ComSee, Pascal Institute, 2005
Institut Pascal
6
Computer Vision: State of the art
Content
>2000: Learning based methods
1.State of the art 2.Why Computer Vision is Challenging ? 3.Related fields 4.Typical tasks of computer vision 5.Computer vision systems 6.Further reading Deep learning, Y. LeCun T. Chateau
Institut Pascal
7
T. Chateau
Institut Pascal
8
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
what about the animal vision system ?
• what about the animal vision system ? • perspec5ve projec5on (from 3D to 2D) • object reflectance • visual features
T. Chateau
Institut Pascal
T. Chateau
9
prey: 360° field of view
predator: stereoscopic vision Institut Pascal
10
Camera Models
Why Computer Vision is Challenging ?
perspec5ve projec5on (from 3D to 2D)
Pinhole model
Image plane Pinhole
Virtual image 11
T. Chateau
Institut Pascal
11
T. Chateau
Institut Pascal
12
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
Scene rendering : the direct problem
exemples of virtual images
Produce an image from parameters
Wikipedia T. Chateau
Institut Pascal
T. Chateau
13
Institut Pascal
14
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
Computer Vision : the inverse problem Es5mate parameters from images
To solve the inverse problem, we need to extract image features
Wikipedia T. Chateau
Wikipedia Institut Pascal
15
T. Chateau
Institut Pascal
16
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
visual features: edges
visual features: shadow
T. Chateau
Institut Pascal
T. Chateau
17
Institut Pascal
18
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
Most popular geometric features used in Computer Vision
visual features: texture
T. Chateau
Institut Pascal
19
T. Chateau
Institut Pascal
20
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
Edges
T. Chateau
Interest Points
Institut Pascal
T. Chateau
21
Institut Pascal
22
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
Should we rely to the human vision ?
But: should we rely on human vision?
T. Chateau
Institut Pascal
23
T. Chateau
Institut Pascal
24
Why Computer Vision is Challenging ?
Why Computer Vision is Challenging ?
Should we rely to the human vision ?
Should we rely to the human vision ? which is the darkest square between A and B?
which is the longest s5ck ? which is the longest table ?
T. Chateau
Institut Pascal
T. Chateau
25
Institut Pascal
26
Why Computer Vision is Challenging ?
Content
Should we rely to the human vision ? which is the darkest square between A and B?
T. Chateau
Institut Pascal
27
1.State of the art 2.Why Computer Vision is Challenging ? 3.Related fields 4.Typical tasks of computer vision 5.Computer vision systems 6.Further reading
T. Chateau
Institut Pascal
28
Computer Vision: Related Fields
Content
1.State of the art 2.Why Computer Vision is Challenging ? 3.Related fields 4.Typical tasks of computer vision 5.Computer vision systems 6.Further reading
(from Wiki) T. Chateau
Institut Pascal
T. Chateau
29
Institut Pascal
30
Typical tasks in Computer Vision
Typical tasks in Computer Vision
Image Processing/Analysis
• Image Processing • Object detec5on and tracking • 2D geometry • 3D geometry for medical applica5ons
T. Chateau
Institut Pascal
31
T. Chateau
Institut Pascal
32
Typical tasks in Computer Vision
Typical tasks in Computer Vision
Image Processing/Analysis
Image Processing/Analysis
original
denoising
T. Chateau
T. Chateau
33
Institut Pascal
34
Typical tasks in Computer Vision
Typical tasks in Computer Vision
Object detec5on and tracking
Object detec5on and tracking
pedestrian detec5on
face detec5on Institut Pascal
35
result
Inpain5ng
Institut Pascal
T. Chateau
mask image
T. Chateau
Institut Pascal
36
Typical tasks in Computer Vision
Typical tasks in Computer Vision
Object detec5on and tracking
Object detec5on and tracking
object detec5on and tracking
object tracking: meanshiU T. Chateau
Institut Pascal
T. Chateau
37
Institut Pascal
Typical tasks in Computer Vision
2D geometry
2D geometry: 2D paVern tracking
V. Lepe5t, EPFL, Lausanne, 2006
Interest point matching
Institut Pascal
39
ComSee, Pascal Institute, 2009 38
Typical tasks in Computer Vision
T. Chateau
MOT: Mul5 objects tracking
T. Chateau
Institut Pascal
40
Typical tasks in Computer Vision
Typical tasks in Computer Vision
2D geometry: video stabiliza5on
before
2D geometry: augmented reality
aUer
Pascal Ins5tute, France, 2006 T. Chateau
ComSee, Pascal Ins5tute, France, 2011 Institut Pascal
T. Chateau
41
Institut Pascal
42
Typical tasks in Computer Vision
Typical tasks in Computer Vision
3D geometry: 3D reconstruc5on
3D geometry 2D interest points
2D/3D optical center
3D model
matching
2D Image
T. Chateau
M. Lhuillier, 2002 Institut Pascal
43
T. Chateau
Institut Pascal
44
Typical tasks in Computer Vision
Typical tasks in Computer Vision
3D geometry: 3D reconstruc5on
3D geometry: 3D reconstruc5on
ComSee, Pascal Ins5tute, 2008 T. Chateau
Institut Pascal
T. Chateau
ComSee Pascal Ins5tute/Cea, France, 2005 Institut Pascal
45
ISPR/ComSee: 3D-Localisation
Monocular based localisa/on for automa/c guidance
46 Visual Memory: Matching
Step 2: 3D reconstruc5on
Step 1: supervised guidance and video recording • Interest point detection (1500 per image) • Correlation ZNCC (11x11 pixels ROI)
image t
47
image t+1
48
Visual memory: interest points detection
Visual Memory: key images selection process
Step 2: 3D reconstruc5on
Step 2: 3D reconstruc5on • Key images: – far enough to produce a precise 3D reconstruction – close enough to keep matching points.
Nb matching > M
N=300, M=400
Nb matching > N
Interest point detection
10
11
49 Visual memory: 3D reconstruction algorithm
Step 2: 3D reconstruc5on Key images
50 Visual memory: 3D reconstruction algorithm
Step 2: 3D reconstruc5on
1 2 3 4 5 6
1 2 3 4 5 6 1 2 3 4
51
3 4 5 6
52
Visual memory: 3D reconstruction algorithm
Visual memory: 3D reconstruction algorithm
Step 2: 3D reconstruc5on
Step 2: 3D reconstruc5on
1 2 3 4 5 6 1 2 3 4 1 2 3
1 2 3 4 5 6
3 4 5 6
1 2 3 4
2 3 4
1 2 3
3 4 5 6
2 3 4
Bundle Adjustment :
3D geometry estimation of the first 3 images
E13
• Essential matrix (5 points algorithm)
estimate
that minimize:
• 3D points reconstruction • Bundle adjustment.
1
2
3
53
54
Visual memory: 3D reconstruction algorithm
Visual memory: 3D reconstruction algorithm
Step 2: 3D reconstruc5on
Step 2: 3D reconstruc5on
1 2 3 4 5 6 1 2 3 4 1 2 3
1 2 3 4 5 6
3 4 5 6
1 2 3 4
2 3 4
1 2 3
3D reconstruction is made for each subset of 3 key images triangulation
…
N-1
55
2 3 4
Fusion step: 3D points
1
3 4 5 6
• based on subsets with common images pose estimation
N
N+1
1
56
2
3
4
Visual Memory: 3D textured points
Visual Memory: 3D points
Step 2: 3D reconstruc5on
Step 2: 3D reconstruc5on
10 m
3D reconstructed points (125 m, 172 key images, 23000 3D image)
(125 m, 172 key images, 23000 3D points) 18
57
19
58
Visual based localization
Processus Innovant : Guidage Temps Réel vidéo
Step 3: real5me online localiza5on Visual memory 3D patchs
2D/3D matching and localization
Camera localization 6 dof
Current image
Realtime localization (15 fps) – precision: 10cm Eric Royer, Maxime Lhuillier, Michel Dhome and Thierry Chateau, Localization in urban environments : monocular vision compared to a differential gps sensor. IEEE CVPR2005, Computer Vision and Pattern Recognition. San Diego, USA, June 2005 20
59
21
60
Expérimentation en milieu urbain : Place de jaude
ISPR/ComSee: 3D-Localisation
Monocular based localisa/on for automa/c guidance Toward the Vipa Project: using two cameras (rear-‐front)
Clermont Fd downtown
23
61
ISPR/ComSee: 3D-Localisation
62
Typical tasks in Computer Vision
Monocular based localisa/on for automa/c guidance
3D geometry: localisa5on, augmented reality
Toward the Vipa Project: using two cameras (rear-‐front)
T. Chateau
63
CVlab, EPFL, Lausanne Institut Pascal
64
Typical tasks in Computer Vision
Typical tasks in Computer Vision
3D geometry: localisa5on, augmented reality,
T. Chateau
ComSee, Pascal Ins5tute/Cea, France, 2013 Institut Pascal
3D geometry: localisa5on, augmented reality,
ComSee, Pascal Ins5tute/Cea, France, 2011 T. Chateau
65
66
Typical tasks in Computer Vision
Typical tasks in Computer Vision And many other applica5ons ...
3D geometry: localisa5on, augmented reality,
T. Chateau
ComSee, Pascal Ins5tute /Cea, France, 2011 Institut Pascal
67
Institut Pascal
T. Chateau
Lasmea/France 68
Institut Pascal
Content
Computer Vision Systems
•Image acquisi5on •Pre-‐processing •Feature extrac5on •Detec5on, segmenta5on •High level processing
1.State of the art 2.Why Computer Vision is Challenging ? 3.Related fields 4.Typical tasks of computer vision 5.Computer vision systems 6.Further reading
T. Chateau
Institut Pascal
T. Chateau
Institut Pascal
69
70
Computer Vision Systems
Computer Vision Systems
Image acquisi5on: sensors
T. Chateau
Institut Pascal
71
Rolling shuVer Vs Global shuVer
T. Chateau
Institut Pascal
72
Computer Vision Systems
Kinect 1 : structured IR active camera
color cameras
T. Chateau
Institut Pascal
73
Kinect 1 : structured IR active camera
75 Institut Pascal
75
Institut Pascal
74
Kinect 1 : structured IR active camera
T. Chateau
74
T. Chateau
76
T. Chateau
Institut Pascal
76
Kinect 2 : time of flight camera
Kinect 2 : time of flight camera
77
T. Chateau
Institut Pascal
77
Kinect 2 : time of flight camera
79 Institut Pascal
79
Institut Pascal
78
Kinect 2 : time of flight camera
T. Chateau
78
T. Chateau
80
T. Chateau
Institut Pascal
80
Kinect 2 : time of flight camera
Kinect 2 : time of flight camera
81
T. Chateau
Institut Pascal
82
T. Chateau
81
Institut Pascal
82
plenoptic cameras
plenoptic cameras
83
T. Chateau
Institut Pascal
83
84
T. Chateau
Institut Pascal
84
plenoptic cameras
plenoptic cameras
85
T. Chateau
Institut Pascal
86
T. Chateau
Institut Pascal
85
86
plenoptic cameras
Computer Vision Systems
Image acquisi5on: communica5on
87
T. Chateau
Institut Pascal
87
T. Chateau
Institut Pascal
88
Computer Vision Systems
Computer Vision Systems
Image processing: hardware
T. Chateau
Image processing: soUware
Institut Pascal
T. Chateau
89
90
Further Readings (from Wiki) ■
Institut Pascal
Further Readings (from Wiki)
Dana H. Ballard and Christopher M. Brown (1982). Computer Vision. Prentice Hall. ISBN 0131653164.
■
Bernd Jähne (2002). Digital Image Processing. Springer. ISBN 3-540-67754-2.
■
David A. Forsyth and Jean Ponce (2003). Computer Vision, A Modern Approach. Prentice Hall. ISBN 0-13-085198-1.
■
David Marr (1982). Vision. W. H. Freeman and Company. ISBN 0-7167-1284-9.
■
Azriel Rosenfeld and Avinash Kak (1982). Digital Picture Processing. Academic Press. ISBN 0-12-597301-2.
■
Richard Hartley and Andrew Zisserman (2003). Multiple View Geometry in Computer Vision. Cambridge University Press. ISBN 0-521-54051-8.
■
Berthold Klaus Paul Horn (1986). Robot Vision. MIT Press. ISBN 0-262-08159-8.
■
■
Olivier Faugeras (1993). Three-Dimensional Computer Vision, A Geometric Viewpoint. MIT Press. ISBN 0-262-06158-9.
Gérard Medioni and Sing Bing Kang (2004). Emerging Topics in Computer Vision. Prentice Hall. ISBN 0-13-101366-1.
■
■
Tony Lindeberg (1994). Scale-Space Theory in Computer Vision. Springer. ISBN 0-7923-9418-6.
Tim Morris (2004). Computer Vision and Image Processing. Palgrave Macmillan. ISBN 0-333-99451-5.
■
James L. Crowley and Henrik I. Christensen (Eds.) (1995). Vision as Process. Springer-Verlag. ISBN 3-540-58143-X and ISBN 0-387-58143-X.
■
E. Roy Davies (2005). Machine Vision : Theory, Algorithms, Practicalities. Morgan Kaufmann. ISBN 0-12-206093-8.
■
Gösta H. Granlund and Hans Knutsson (1995). Signal Processing for Computer Vision. Kluwer Academic Publisher. ISBN 0-7923-9530-1.
■
R. Fisher, K Dawson-Howe, A. Fitzgibbon, C. Robertson, E. Trucco (2005). Dictionary of Computer Vision and Image Processing. John Wiley. ISBN 0-470-01526-8.
■
Reinhard Klette, Karsten Schluens and Andreas Koschan (1998). Computer Vision - ThreeDimensional Data from Images. Springer, Singapore. ISBN 981-3083-71-9.
■
Nikos Paragios and Yunmei Chen and Olivier Faugeras (2005). Handbook of Mathematical Models in Computer Vision. Springer. ISBN 0-387-26371-3.
■
Emanuele Trucco and Alessandro Verri (1998). Introductory Techniques for 3-D Computer Vision. Prentice Hall. ISBN 0132611082.
■
Wilhelm Burger and Mark J. Burge (2007). Digital Image Processing: An Algorithmic Approach Using Java. Springer. ISBN 1846283795 and ISBN 3540309403.
■
Milan Sonka, Vaclav Hlavac and Roger Boyle (1999). Image Processing, Analysis, and Machine Vision. PWS Publishing. ISBN 0-534-95393-X.
■
Pedram Azad, Tilo Gockel, Rüdiger Dillmann (2008). Computer Vision - Principles and Practice. Elektor International Media BV. ISBN 0905705718.
■
Bernd Jähne and Horst Haußecker (2000). Computer Vision and Applications, A Guide for Students and Practitioners. Academic Press. ISBN 0-12-379777-2.
■
Linda G. Shapiro and George C. Stockman (2001). Computer Vision. Prentice Hall. ISBN 0-13-030796-3.
T. Chateau
Institut Pascal
91
T. Chateau
Institut Pascal
92
