Joint Acquisition of RGB and NIR Images with the Bayer CFA ! !
Zahra Sadeghipoor, Sabine Süsstrunk
EPFL-IC-IVRG
!
December 4, 2013
Fundamental Imaging Problem • What is the contribu7on to the pixel value of the object reflectance versus the light source?
?
Fundamental Imaging Problem • What is the contribu7on to the pixel value of the object reflectance versus the light source?
? or A red surface under white light?
A white surface under red light?
Fundamental Imaging Problem • What is the contribu7on to the pixel value of the object reflectance versus the light source?
Two shades of )le under one light? One shade of )le under two lights?
Image Forma7on Camera Response
Light Power E(λ)
wavelength λ
Surface Reflectance S(λ)
wavelength λ
Quantum Efficiency Q(λ)
wavelength λ
Scene Understanding •Separa7ng the effect of illuminant and
surface reflectance of the camera response (pixel value) is an ill-‐posed problem.
Scene Understanding •Separa7ng the effect of illuminant and
surface reflectance of the camera response (pixel value) is an ill-‐posed problem. •To es7mate the two, we either make assump7ons about the scene…
Scene Understanding •Separa7ng the effect of illuminant and
surface reflectance of the camera response (pixel value) is an ill-‐posed problem. •To es7mate the two, we either make assump7ons about the scene… •...or we capture more informa7on about the scene.
RGB Visible Spectrum 400 nm
Near-Infrared 700 nm
Visible (Red, Green, and Blue: RGB)
Near Infrared (NIR) Visible Spectrum
Near-Infrared 700 nm
1100 nm
Near-infrared (NIR)
RGB + Near Infrared Visible Spectrum 400 nm
Near-Infrared 700 nm
1100 nm
+ Visible (Red, Green, and Blue: RGB)
Near-infrared (NIR)
Benefits of RGB + NIR •We can exploit the correla7on and de-‐ correla7on of visible and NIR image frequencies and intensi7es to:
-‐ enhance the visual quality of images/videos. -‐ extract more accurate informa7on about the scene.
Human vision and the spectrum
Human vision and the spectrum
The sensi7vity of silicon
Camera Design
Bayer Color Filter Array (CFA)
Camera Sensi7vity
Bayer Color Filter Array (CFA)
Camera Sensi7vity
Near Infrared Blocking Filter (Hot Mirror)
All digital cameras are inherently sensi7ve to NIR…
All digital cameras are inherently sensi7ve to NIR… …if we remove the hot mirror!
Camera construc7on
C. Fredembach and S. Süsstrunk, Colouring the near infrared, IS&T/SID 16th Color Imaging Conference, 2008. MERL Best Student Paper Award
Camera construc7on
…or you buy one on-‐line!
C. Fredembach and S. Süsstrunk, Colouring the near infrared, IS&T/SID 16th Color Imaging Conference, 2008. MERL Best Student Paper Award
Capturing visible images
Visible = Image
Visible Image
Capturing NIR images
NIR = Image
NIR Image
Haze Removal: RGB
L. Schaul, C. Fredembach, and S. Süsstrunk, Color image de-‐hazing using the near-‐infrared, Proc. IEEE Interna7onal Conference on Image Processing (ICIP), 2009.
Haze Removal: RGB+NIR
L. Schaul, C. Fredembach, and S. Süsstrunk, Color image de-‐hazing using the near-‐infrared, Proc. IEEE Interna7onal Conference on Image Processing (ICIP), 2009.
Skin Smoothing
Visible
NIR
Visible + NIR
C. Fredembach, N. Barbuscia, and S. Süsstrunk, Combining visible and near-‐infrared images for realis7c skin smoothing, Proc. IS&T/SID 17th Color Imaging Conference, 2009.
Skin Smoothing
C. Fredembach, N. Barbuscia, and S. Süsstrunk, Combining visible and near-‐infrared images for realis7c skin smoothing, Proc. IS&T/SID 17th Color Imaging Conference, 2009.
Camera Museum in Vevey
S. Süsstrunk, C. Fredembach, and D. Tamburrino, Automa7c skin enhancement with visible and near-‐infrared image fusion, Proc. ACM Mul7media, 2010. Best Demo Paper Award
Camera Museum in Vevey Visible
Visible + NIR
Preference: 25.4% (1,235)
Preference: 74.6% (3,623)
S. Süsstrunk, C. Fredembach, and D. Tamburrino, Automa7c skin enhancement with visible and near-‐infrared image fusion, Proc. ACM Mul7media, 2010. Best Demo Paper Award
Shadow Detec7on
Visible
NIR
Binary Shadow Mask
D. Rüfenacht, C. Fredembach and S. Süsstrunk, Automatic and accurate shadow detection using nearinfrared information, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.
Image Segmenta7on
Visible
Intrinsic Image
Visible Segmenta1on
Visible and NIR Segmenta1on
N. Salama7 and S. Süsstrunk, Material-‐Based Object Segmenta7on Using NIR Informa7on, Proc. IS&T/SID 18th Color Imaging Conference, 2010. Best Interac7ve Paper Award
Scene Categoriza7on
M. Brown and S. Süsstrunk, Mul7-‐spectral SIFT for Scene Category Recogni7on, CVPR 2011.
Seman7c Image Segmenta7on
Visible
NIR
V Seg
V+NIR Seg
GT
N. Salama7, D. Larlus, G. Csurka, and S. Süsstrunk, Seman7c Image Segmenta7on Using Visible and Near-‐ Infrared Channels, in ECCV’s 4th Color and Photometry in Computer Vision Workshop (2012).
Current Research • Camera with RGB + NIR capture on a single sensor.
Design Criteria • Single sensor.
• Single shot.
• Similar to current cameras. 4
Optimized CFA RGB+NIR Camera Pipeline
Reconstruction Demosaicing
[Lu et al. 2009] [Sadeghipoor et al. 2011]
5
Bayer CFA for Joint Acquisition RGB+NIR Camera Pipeline
Reconstruction Demosaicing
[Sadeghipoor et al. 2013]
6
Bayer CFA for Joint Acquisition RGB+NIR Camera Pipeline
?
Reconstruction Demosaicing
6
Problem Formulation scene
Visible
NIR
7
Problem Formulation scene
7
Problem Formulation scene
separation
7
Problem Formulation scene
separation
color
demosaicing
7
Problem Formulation scene
separation
color
demosaicing
number of equations is less than the number of unknowns!
Unique solution?
7
Solution Compressive sensing tools [Donoho06]
8
Solution Compressive sensing tools [Donoho06] Main idea:
Use prior knowledge about the target signal as
additional equations.
8
Solution Compressive sensing tools [Donoho06] Main idea:
Use prior knowledge about the target signal as
additional equations. Main assumption in CS:
Sparsity of signals in a specific transform domain. 8
Proposed Algorithm measurements Green and NIR separation NIR interpolation Red/Blue and NIR separation 9
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
10
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
10
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
10
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
10
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
10
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
11
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
11
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
11
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
12
Green and NIR Separation measurements y1 =! G+NIR
y2 =! G+NIR y3 =! G+NIR
y4 =! G+NIR
y5 =! G+NIR
unknowns NIR1
G1
NIR2 NIRG2 3
G NIR4 3 G4
NIR5
G5
12
Results Original
Customized CFA
Bayer CFA
13
Results Original
Customized CFA
Bayer CFA
14
Summary • Motivation:
• Applications of RGB and NIR images.
• Potential of current color cameras for NIR acquisition.
• Proposed algorithm:
• Using compressive sensing tools.
• Exploring the target signals structure in a sparsifying transform.
!
15
Thank you! Z. Sadeghipoor, Y.M. Lu, and S. Süsstrunk, “A Novel Compressive
Sensing Approach to Simultaneously Acquire Color and Near-Infrared
Images on a Single Sensor,” Proc. ICASSP, 2013.
http://ivrg.epfl.ch/publications
