Bayesian View Synthesis and Image-Based Rendering Principles
1
1
Sergi Pujades, Frédéric Devernay, Bastian Goldluecke CVPR 2014 1
2 University of Konstanz
2
Image Based Rendering
Input views
Input views
v2 v1 INRIA Grenoble, France
2
CVPR 2014 - 27 June 2014
Image Based Rendering
Input views
Input views
v2
? v1 u INRIA Grenoble, France
2
Target view CVPR 2014 - 27 June 2014
Image Based Rendering Scene Geometry
Input views
Input views
v2
? v1 u INRIA Grenoble, France
2
Target view CVPR 2014 - 27 June 2014
Image Based Rendering Scene Geometry
Input views
Input views
v2
x v1 u INRIA Grenoble, France
2
Target view CVPR 2014 - 27 June 2014
Image Based Rendering Scene Geometry
Input views
Input views
v2
x v1 u INRIA Grenoble, France
2
Target view CVPR 2014 - 27 June 2014
Image Based Rendering Scene Geometry
Input views
Input views
v2
x v1 u INRIA Grenoble, France
2
Target view CVPR 2014 - 27 June 2014
State of the art
IBR Continum
Scene Geometry less Light field
INRIA Grenoble, France
more Lumigraph
3
Texture-mapped models
CVPR 2014 - 27 June 2014
State of the art Unstructured Lumigraph Rendering C. Buehler et al. - SIGGRAPH 2001 8 Desirable Properties
• Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Continuity • Real-time
INRIA Grenoble, France
4
CVPR 2014 - 27 June 2014
State of the art Unstructured Lumigraph Rendering C. Buehler et al. - SIGGRAPH 2001 8 Desirable Properties
• Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Continuity • Real-time
INRIA Grenoble, France
4
CVPR 2014 - 27 June 2014
Minimal angular deviation
Input views
Input views
v2 v1 INRIA Grenoble, France
5
CVPR 2014 - 27 June 2014
Minimal angular deviation
Input views
Input views
v2 v1 INRIA Grenoble, France
u Target view 5
CVPR 2014 - 27 June 2014
Minimal angular deviation
Input views
Input views
v2 v1 INRIA Grenoble, France
u Target view 5
CVPR 2014 - 27 June 2014
Minimal angular deviation
Input views
Input views
v2 v1 INRIA Grenoble, France
u Target view 5
CVPR 2014 - 27 June 2014
Resolution Sensitivity
Input views
v2
v2
v1
Input views v2 v1
INRIA Grenoble, France
6
CVPR 2014 - 27 June 2014
Resolution Sensitivity
Input views
v2 v1
Input views
v2
u
Target view v2
v1
INRIA Grenoble, France
6
CVPR 2014 - 27 June 2014
Resolution Sensitivity
Input views
v2 v1
Input views
v2
u
Target view v2
v1
INRIA Grenoble, France
6
CVPR 2014 - 27 June 2014
Resolution Sensitivity
Input views
v2 v1
Input views
v2
u
Target view v2
v1
INRIA Grenoble, France
6
CVPR 2014 - 27 June 2014
State of the art limitations
For both properties: • Minimal angular deviation • Resolution sensitivity
No formal deduction of heuristics Manual parameter tuning depending on the scene
INRIA Grenoble, France
7
CVPR 2014 - 27 June 2014
New properties proposed • Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Formal deduction of heuristics • Physics-based parameters • Continuity • Real-time
INRIA Grenoble, France
8
CVPR 2014 - 27 June 2014
New properties proposed • Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Formal deduction of heuristics • Physics-based parameters • Continuity • Real-time
INRIA Grenoble, France
8
CVPR 2014 - 27 June 2014
State of the art Method
Formal deduction Physics-Based Parameters
Resolution sensitivity
Minimal angular deviation
Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering
Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy
Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields
Our method CVPR 2014
INRIA Grenoble, France
9
CVPR 2014 - 27 June 2014
State of the art Method
Formal deduction Physics-Based Parameters
Resolution sensitivity
Minimal angular deviation
Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering
Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy
Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields
Our method CVPR 2014
INRIA Grenoble, France
9
CVPR 2014 - 27 June 2014
State of the art Method
Formal deduction Physics-Based Parameters
Resolution sensitivity
Minimal angular deviation
Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering
Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy
Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields
Our method CVPR 2014
INRIA Grenoble, France
9
CVPR 2014 - 27 June 2014
State of the art Method
Formal deduction Physics-Based Parameters
Resolution sensitivity
Minimal angular deviation
Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering
Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy
Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields
Our method CVPR 2014
INRIA Grenoble, France
9
CVPR 2014 - 27 June 2014
State of the art Method
Formal deduction Physics-Based Parameters
Resolution sensitivity
Minimal angular deviation
Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering
Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy
Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields
Our method CVPR 2014
INRIA Grenoble, France
9
CVPR 2014 - 27 June 2014
State of the art Method
Formal deduction Physics-Based Parameters
Resolution sensitivity
Minimal angular deviation
Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering
Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy
Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields
Our method CVPR 2014
INRIA Grenoble, France
9
CVPR 2014 - 27 June 2014
State of the art Method
Formal deduction Physics-Based Parameters
Resolution sensitivity
Minimal angular deviation
Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering
Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy
Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields
Our method CVPR 2014
INRIA Grenoble, France
9
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem
v2
? v1
Input view
INRIA Grenoble, France
u
Target view
10
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem
? v1
Input view
INRIA Grenoble, France
v2
? u
Target view
10
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem
x v1
Input view
INRIA Grenoble, France
v2
u
Target view
11
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Scene Geometry
x v1
Input view
INRIA Grenoble, France
v2
u
Target view
11
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Scene Geometry
x v1
Input view
INRIA Grenoble, France
⌧˜i
v2
u
Target view
11
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
x v1
Input view
INRIA Grenoble, France
v˜i (x) = (u
⌧˜i
⌧˜i )(x)
v2
u
Target view
11
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
x v1
Input view
INRIA Grenoble, France
v˜i (x) = (u
⌧˜i )(x)
Generative Model Perfect image formation v2 description
⌧˜i u
Target view
11
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
x v1
Input view
INRIA Grenoble, France
v˜i (x) = (u
⌧˜i )(x)
Generative Model Perfect image formation v2 description
⌧˜i u
assuming Lambertian model
Target view
11
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
x v1
Input view
INRIA Grenoble, France
v˜i (x) = (u
⌧˜i
⌧˜i )(x)
v2
u
Target view
12
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
v˜i (x) = (u
⌧˜i )(x)
Observed image
vi (x) = v˜i (x) + es (x) x v1
Input view
INRIA Grenoble, France
⌧˜i
v2
u
Target view
12
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
v˜i (x) = (u
⌧˜i )(x)
Observed image
vi (x) = v˜i (x) + es (x) x v1
Input view
INRIA Grenoble, France
⌧˜i
v2
u
Target view
12
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
v˜i (x) = (u Observed image
⌧˜i )(x) Sensor noise
vi (x) = v˜i (x) + es (x) x v1
Input view
INRIA Grenoble, France
⌧˜i
v2
u
Target view
12
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
v˜i (x) = (u Observed image
⌧˜i )(x) Sensor noise
vi (x) = v˜i (x) + es (x) x v1
Input view
INRIA Grenoble, France
⌧˜i
v2
Gaussian distribution u
Target view
12
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Perfect image
Scene Geometry
v˜i (x) = (u Observed image
⌧˜i )(x) Sensor noise
vi (x) = v˜i (x) + es (x) x v1
Input view
⌧˜i
v2
Gaussian distribution u
Target view Least squares minimisation problem
INRIA Grenoble, France
12
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Spatial and Angular Variational Super-resolution of 4D Light Fields
Scene Geometry
S. Wanner and B. Goldluecke ECCV 2012
vi (x) = v˜i (x) + es (x) x v1
Input view
INRIA Grenoble, France
⌧˜i
v2
u
Target view
Physics based Resolution sensibility Minimal angular deviation
13
CVPR 2014 - 27 June 2014
Bayesian Approach: Inverse Problem Spatial and Angular Variational Super-resolution of 4D Light Fields
Scene Geometry
S. Wanner and B. Goldluecke ECCV 2012
vi (x) = v˜i (x) + es (x) x v1
Input view
⌧˜i
v2
u
Target view
Physics based Resolution sensibility Minimal angular deviation
WHY? INRIA Grenoble, France
13
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Scene Geometry
x v1
Input view
INRIA Grenoble, France
v2
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
x v1
Input view
INRIA Grenoble, France
v2
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
Depth distribution
x v1
Input view
INRIA Grenoble, France
v2
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
Depth distribution
x v1
Input view
INRIA Grenoble, France
⌧˜i
v2
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
Depth distribution
x v1
Input view
INRIA Grenoble, France
v2
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
Observed image
Depth distribution
vi (x) = v˜i (x) + es (x) + eg (x)
x v1
Input view
INRIA Grenoble, France
v2
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
Observed image
Depth distribution
vi (x) = v˜i (x) + es (x) + eg (x)
x v1
Input view
INRIA Grenoble, France
v2 Perfect image
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
Observed image
Depth distribution
vi (x) = v˜i (x) + es (x) + eg (x)
x v1
Input view
INRIA Grenoble, France
Sensor noise
v2 Perfect image
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
Observed image
Depth distribution
vi (x) = v˜i (x) + es (x) + eg (x)
x v1
Input view
INRIA Grenoble, France
Sensor noise
v2 Perfect image
Geometric Noise
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
details in the paper Observed image
Depth distribution
vi (x) = v˜i (x) + es (x) + eg (x)
x v1
Input view
INRIA Grenoble, France
Sensor noise
v2 Perfect image
Geometric Noise
u
Target view
14
CVPR 2014 - 27 June 2014
Bayesian Approach: Proposed Method Geometric uncertainty
details in the paper Observed image
Depth distribution
vi (x) = v˜i (x) + es (x) + eg (x)
x v1
Input view
INRIA Grenoble, France
Sensor noise
v2 Perfect image
Geometric Noise
Support of the projected Gaussian depends on the angular deviation
u
Target view
14
CVPR 2014 - 27 June 2014
Deducted weights
|det D⌧i |
1
2 zi
✓
@ (u ⌧i ) @zi
◆2
1
Minimal angular deviation Physics based Resolution sensitivity
INRIA Grenoble, France
15
CVPR 2014 - 27 June 2014
Deducted weights
|det D⌧i |
1
2 zi
✓
@ (u ⌧i ) @zi
◆2
1
Minimal angular deviation Physics based Resolution sensitivity
INRIA Grenoble, France
15
CVPR 2014 - 27 June 2014
Deducted weights
|det D⌧i |
1
2 zi
✓
@ (u ⌧i ) @zi
◆2
1
Minimal angular deviation Physics based Resolution sensitivity
INRIA Grenoble, France
15
CVPR 2014 - 27 June 2014
Deducted weights
|det D⌧i |
1
2 zi
✓
@ (u ⌧i ) @zi
◆2
1
Minimal angular deviation Physics based Resolution sensitivity
Weighting factor depends on correspondence confidence
INRIA Grenoble, France
15
CVPR 2014 - 27 June 2014
Deducted weights
|det D⌧i |
1
2 zi
✓
@ (u ⌧i ) @zi
◆2
1
Minimal angular deviation Physics based Resolution sensitivity
Weighting factor depends on correspondence confidence
INRIA Grenoble, France
15
CVPR 2014 - 27 June 2014
Deducted weights
|det D⌧i |
1
2 zi
✓
@ (u ⌧i ) @zi
◆2
1
Minimal angular deviation Physics based Resolution sensitivity
Weighting factor depends on
correspondence confidence image content (color gradient along epipolar line) INRIA Grenoble, France
15
CVPR 2014 - 27 June 2014
Experiments Implementation of a simplified camera configuration 4D Light Field
Stanford multi-camera array
INRIA Grenoble, France
16
CVPR 2014 - 27 June 2014
The (New) Stanford Light Field Archive Tarot
INRIA Grenoble, France
Truck
17
CVPR 2014 - 27 June 2014
HCI Lightfield Dataset Maria
INRIA Grenoble, France
Still Life
18
CVPR 2014 - 27 June 2014
?
INRIA Grenoble, France
19
CVPR 2014 - 27 June 2014
Results
Ground truth
Previous method Wanner and Goldluecke ECCV 2012
INRIA Grenoble, France
20
CVPR 2014 - 27 June 2014
Results
Ground truth
Previous method Wanner and Goldluecke ECCV 2012
INRIA Grenoble, France
20
CVPR 2014 - 27 June 2014
Results
Ground truth
Previous method
Proposed method
Wanner and Goldluecke ECCV 2012
INRIA Grenoble, France
20
CVPR 2014 - 27 June 2014
What is happening? Better selection of the contributing views based on : • View distance • color gradient aligned with view displacement
INRIA Grenoble, France
21
CVPR 2014 - 27 June 2014
What is happening? Better selection of the contributing views based on : • View distance • color gradient aligned with view displacement
INRIA Grenoble, France
21
CVPR 2014 - 27 June 2014
What is happening? Better selection of the contributing views based on : • View distance • color gradient aligned with view displacement
INRIA Grenoble, France
21
CVPR 2014 - 27 June 2014
What is happening? Better selection of the contributing views based on : • View distance • color gradient aligned with view displacement
INRIA Grenoble, France
22
CVPR 2014 - 27 June 2014
What is happening? Better selection of the contributing views based on : • View distance • color gradient aligned with view displacement
INRIA Grenoble, France
23
CVPR 2014 - 27 June 2014
What is happening? Better selection of the contributing views based on : • View distance • color gradient aligned with view displacement
INRIA Grenoble, France
24
CVPR 2014 - 27 June 2014
Results
Ground truth
Previous method
Proposed method
Wanner and Goldluecke ECCV 2012
INRIA Grenoble, France
25
CVPR 2014 - 27 June 2014
Status and future work • Use of geometric proxies • Unstructured input • Epipole consistency • Equivalent ray consistency • Minimal angular deviation • Resolution sensitivity • Formal deduction • Physics-based parameters • Continuity • Real-time
INRIA Grenoble, France
26
CVPR 2014 - 27 June 2014
Status and future work • Use of geometric proxies • Unstructured input • Epipole consistency • Equivalent ray consistency • Minimal angular deviation • Resolution sensitivity • Formal deduction • Physics-based parameters • Continuity • Real-time
INRIA Grenoble, France
26
CVPR 2014 - 27 June 2014
Conclusion New generative model for IBR Unify current knowledge Improve results Code available as part of cocolib library http://sourceforge.net/projects/cocolib/
INRIA Grenoble, France
27
CVPR 2014 - 27 June 2014
Take home messages
Bayesian formulation: Use physically-sound parameters! Uncertainty is helpful: Don’t throw away your covariance matrices!
INRIA Grenoble, France
28
CVPR 2014 - 27 June 2014
Bayesian View Synthesis and Image-Based Rendering Principles
1
1
Sergi Pujades, Frédéric Devernay, Bastian Goldluecke CVPR 2014 1
2 University of Konstanz
2
