Focus Mismatch Detection in stereoscopic content Frédéric Devernay, Sergi Pujades and Vijay Ch.A.V. INRIA Grenoble, France
Stereoscopic Displays and Applications 2012
Motivation: stereoscopic visual quality For best visual quality and visual comfort stereoscopic image pairs should:
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
2
Motivation: stereoscopic visual quality For best visual quality and visual comfort stereoscopic image pairs should: 1. be geometrically aligned
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
2
Motivation: stereoscopic visual quality For best visual quality and visual comfort stereoscopic image pairs should: 1. be geometrically aligned 2. be color-balanced
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
2
Motivation: stereoscopic visual quality For best visual quality and visual comfort stereoscopic image pairs should: 1. be geometrically aligned 2. be color-balanced 3. have same depth of field and focus distance
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
2
Motivation: stereoscopic visual quality For best visual quality and visual comfort stereoscopic image pairs should: 1. be geometrically aligned 2. be color-balanced
solved by post-processing
3. have same depth of field and focus distance
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
2
Motivation: stereoscopic visual quality For best visual quality and visual comfort stereoscopic image pairs should: 1. be geometrically aligned 2. be color-balanced
solved by post-processing
3. have same depth of field and focus distance post-processing would degrade image quality
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
2
Motivation: stereoscopic visual quality For best visual quality and visual comfort stereoscopic image pairs should: 1. be geometrically aligned 2. be color-balanced
solved by post-processing
3. have same depth of field and focus distance post-processing would degrade image quality avoid it: detect while shooting Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
2
Our goal Given a stereoscopic pair of images, we want to answer :
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
3
Our goal Given a stereoscopic pair of images, we want to answer : • Are focal distances and depth of field the same?
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
3
Our goal Given a stereoscopic pair of images, we want to answer : • Are focal distances and depth of field the same? • Which manual adjustment can solve it: aperture? focus?
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
3
Our goal Given a stereoscopic pair of images, we want to answer : • Are focal distances and depth of field the same? • Which manual adjustment can solve it: aperture? focus?
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
3
Our goal Given a stereoscopic pair of images, we want to answer : • Are focal distances and depth of field the same? • Which manual adjustment can solve it: aperture? focus?
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
3
Our goal Given a stereoscopic pair of images, we want to answer : • Are focal distances and depth of field the same? • Which manual adjustment can solve it: aperture? focus?
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
3
Method outline 1. Detecting per-pixel focus mismatch. 2. Give feedback to operator
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
4
Detecting per-pixel focus mismatch
1
Detecting per-pixel focus mismatch
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
5
Focal Blur Model Assuming parallel (or near parallel) cameras, optics imply: • Stereo disparity depends on depth. • Focal blur size is linear with the stereo disparity. (Rajagopalan et al. 2004, Schechner et al. 1988)
focus distance disparity
In Focus 1 Out of Focus
depth of field focal blur size
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
6
Focal Blur Model Assuming parallel (or near parallel) cameras, optics imply: • Stereo disparity depends on depth. • Focal blur size is linear with the stereo disparity. (Rajagopalan et al. 2004, Schechner et al. 1988)
focus distance disparity
2 parameters
In Focus 1 Out of Focus
depth of field focal blur size
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
6
Measuring focal blur size is ill-posed All-in-focus image
Observed image
Focal ⨂ Blur Size =
We would like to measure focal blur size We only have access to the observed images Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
7
Measuring focal blur size is ill-posed All-in-focus image
Observed image
Focal ⨂ Blur Size =
We would like to measure focal blur size We only have access to the observed images Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
7
Measuring focal blur size is ill-posed All-in-focus image
Observed image
Focal ⨂ Blur Size =
Problem: non-textured scene We would like to measure focal blur size We only have access to the observed images Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
7
Measuring focal blur difference is possible
- More focal blur causes more image blur - Less focal blur causes less image blur
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
8
Measuring focal blur difference is possible
- More focal blur causes more image blur - Less focal blur causes less image blur
The sign of focal blur difference is the same as the sign of image blur difference.
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
8
Focal Blur Difference
Difference
d Left Focus Model Right Focus Model
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
9
Sign of Focal and Image Blur Difference Sign of Difference Difference
d Left Focus Model Right Focus Model
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
10
Sign of Focal and Image Blur Difference Sign of Difference Difference
d Left Focus Model Right Focus Model
Reminder: Sign of focal blur size difference = Sign of image blur difference. Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
10
All Configurations: Focal Blur Size Difference DOFl < DOFr
FDl < FDr
FDl = FDr
d
FDl > FDr
d
d
Legend :
DOFl = DOFr
d
d
d
DOFl > DOFr
Left focal b
d
d
d
Right focal
Focal blur d
Figure 2. Graphs of the focal blur size and the di↵erence of left and right focal blur. Each cell contain left and right focal blur functions, in red and blue respectively, corresponding to the given parameters. Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection” Stereoscopic Displays and Applications 2012 11 both functions is shown in green.
Figure 2. Graphs of the focal blur size and the di↵erence of left and right focal blur. Each cell contains the g left and right focal blur functions, in red and blue respectively, corresponding to the given parameters. The d both functions is shown in green.
All Configurations: Sign of Focal/Image Blur Difference DOFl < DOFr
FDl < FDr
FDl = FDr
d
L = ( , +, )
FDl > FDr
d
L=( , )
d
L = ( , +, )
Legend :
DOFl = DOFr
Left focal blur Right focal blur Focal blur di↵erence d
DOFl > DOFr
L = (+, )
d
L = ()
d
L = (+, , +)
d
L = ( , +)
d
L = (+, +)
Sign of Focal blur di↵ere
d
L = (+, , +)
Figure 3. Sign of the di↵erence between left and right focal blur (in orange), and left and right focal blur functi red and blue lines). TheCh.A.V. sign of“Focus the di↵erence right image blur should give the2012 same 12 resu Devernay, Pujades, Mismatch between Detection”the left andStereoscopic Displays and Applications
Algorithm Outline
From two images: Compute sign of image blur difference
→
The curve shape gives the focus configuration.
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
13
Image blur measurement: state of the art Depth from focus: “Given N images of one object with known different focus distances, compute depth.” For each pixel decide which image is more in focus. Depth from defocus: “Given two images of one object with known different apertures, compute depth.” For each pixel quantify the focus difference. In our case: Focus mismatch detection : detect a difference.
Photostereosynthesis Lumière, 1920. Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
14
Image blur measurement: state of the art Depth from focus: “Given N images of one object with known different focus distances, compute depth.” For each pixel decide which image is more in focus. Depth from defocus: “Given two images of one object with known different apertures, compute depth.” For each pixel quantify the focus difference. In our case: Focus mismatch detection : detect a difference.
Photostereosynthesis Lumière, 1920.
use Depth from focus tools
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
14
ed a disparity map computation which preforms well even with reduce We thus use a real-time multi-scale method14 which finds good dispa Image blurcompute measurement : areas by left-right con red textures. We also semi-occluded Sum of Modified lowing computation. Let d(i) beLaplacian the left-to-right disparity of pixel i = ) = (xl + d(i), y) in the right image. From Nayar & Nakagawa, 1994 “Depth from focus”
surement Modified Laplacian at a pixel: captures “textureness”
2 mage r blur of corresponding I(x points 1, y)from I(xthe + 1, left y)| +and right images w M L I(x, y) = |2I(x, y) Laplacian)15, 16 which was primarily designed for depth-from-focus ap
|2I(x, y)
I(x
1, y)
I(x + 1, y)|
r2M L I(i, j), for r2M LSum I(i,ofj)Modified T , Laplacian y) = |2I(x, y)
I(xSML(i) s, y)=
x+1 X
y+1 X
2 2 r I(i, j), for r I(i, j) M L I(x + s, y)| + |2I(x, y) M LI(x, y
i=x 1 j=y 1
T s)
I
discards hreshold value, N isThreshold the window sizesensor used noise: for the computation of SM nt, these were set respectively to T = 5 (for 8-bits images), N = 1 a mage, SMLl (i) is the SML operator computed at this pixel, and SMLr Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
15
the most question: there a focus mismatch. From the5 shape of th step size.important In our experiment, these iswere set respectively to T = (for 8-bits that the DOF bethf l > l = DOF r . lHowever, l = DOF r couldat pixelFD i= (x,FD y)rinand theDOF left image, SML (i) is the SML operator computed Image difference measurement: incomplete model: blur DOF l > DOF l < DOFr are possible candidates, t at the corresponding pixel in ther and rightDOF image. Sign of12left and right SML image difference Finally, in Fig. we present the obtained zebras on the original images for Let M (i) be the sign of the di↵erence of SML between two pixels: is focused on the tree in the back, right is focused on the fountain). We can ob each image have Mapping frombeen left precisely to right: marked. M (i) = sign (SMLl (i) SMLr (i)) .
Dense disparity map.
dmin
dmax
M (i) < 0
M (i) = 0
M (i) > 0
Min
Figure 10. Intermediate results for the (LFar,RNear) case. leftNear to right: (a Example:test Left FarFrom - Right Di↵erence: M (i), (c) Max of SML: w(i). The blue and green dots in the disparity ma the Near Devernay, and Far sample images. Pujades, Ch.A.V. “Focus Mismatch Detection” Stereoscopic Displays and Applications 2012 16
imum and maximum values of C(d) are very close to the actual values of the hese disparities there is an important focus mismatch. We can easily answer e is a focus mismatch. From the shape of the model we could also conclude Fr . However, the DOFl = DOFr could be false if we had only access to an and DOFl < DOFr are possible candidates, too.
Mean Sign of SML difference wrt. disparity
he obtained zebras on the original images for our (LFar,RNear) example (left right is focused on the fountain). We can observe that the correct regions of wrt: with respect to arked.
x
Mean Sign +1
M (i) < 0
M (i) = 0
M (i) > 0
→0 Min w(i)
Disparity Max w(i)
he (LFar,RNear) test case. From left to right: (a) Disparity: d(i), (b) Sign of SML (i). The blue and green dots in the disparity map correspond to focus distances for
on
-1
we first tested our algorithm using the ground-truth disparity map of the scene e five images for each view. The 25 obtained results are presented in Fig. 13. following questions, sorted by order of difficulty:
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
17
Finding a simple blur model From the Mean sign curve we find simple blur model
+1
0
Mean Sign
+1
→
Mean Sign
0
Disparity
Disparity -1
-1
details in the paper Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
18
Section 1 summary
From two images: Compute sign of image blur difference
→
The curve shape gives the focus configuration.
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
19
2
Give feedback to operator
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
20
Zebras on images +1 0
Mean Sign
Disparity
⨁
→
-1
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
21
Zebras on images +1 0
Mean Sign
Disparity
⨁
→
-1
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
21
Manual adjustement hints • Answers to the questions: • Are both focal distances and depth of field perfectly matched? • Are both focal distances equal? Which one is bigger? • Are both depths-of-field equal? Which one is bigger?
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
22
Manual adjustement hints • Answers to the questions: • Are both focal distances and depth of field perfectly matched? • Are both focal distances equal? Which one is bigger? • Are both depths-of-field equal? Which one is bigger? • Obtained by shape classification
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
22
Manual adjustement hints • Answers to the questions: • Are both focal distances and depth of field perfectly matched? • Are both focal distances equal? Which one is bigger? • Are both depths-of-field equal? Which one is bigger? • Obtained by shape classification • Sometimes, some questions cannot be answered
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
22
Manual adjustement hints • Answers to the questions: • Are both focal distances and depth of field perfectly matched? • Are both focal distances equal? Which one is bigger? • Are both depths-of-field equal? Which one is bigger? • Obtained by shape classification • Sometimes, some questions cannot be answered • details in the paper
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
22
3
CONCLUSION
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
23
Conclusion and Future Work Conclusion • We presented a novel method to detect focus mismatch • Evaluation on synthetic data is very promising: (details in the paper) • “Are both focus distances and depth of field perfectly matched?” • 100% accuracy • Always capable of accurately telling at least which camera is less in focus. • All steps run in real-time Future Work • Validate proposed method with real footage from actual cameras. • Quantification of the differences. • Detect astigmatism? A non-flat mirror in a mirror rig?
Devernay, Pujades, Ch.A.V. “Focus Mismatch Detection”
Stereoscopic Displays and Applications 2012
24
Focus Mismatch Detection in stereoscopic content Frédéric Devernay, Sergi Pujades and Vijay Ch.A.V.
Thank you. INRIA - Grenoble
www.inria.fr
This work was done within the 3DLive project supported by the French Ministry of Industry http://3dlive-project.com/
