Incremental-LDI for Multi-View Coding Generation & Representation
Vincent JANTET IRISA Rennes - TEMICS Team FRANCE
Supervisors: Christine Guillemot - IRISA Luce Morin - IETR - INSA Gaël Sourimant - IRISA
April 1, 2010
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
1 / 19
Context Multi-view videos Desired functionalities: 3DTV: Depth feeling by stereovision simulation.
Figure: Multi-view acquisition
FVV: (for Free Viewpoint Video) Live viewpoint selection.
Problems Acquisition: Synchronization, calibration. . . Compression: Compact representation of the huge amount of data. Rendering: Photo-realistic virtual view generation. Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
Figure: 3D rendering April 1, 2010
2 / 19
Context Multi-view videos Desired functionalities: 3DTV: Depth feeling by stereovision simulation.
Figure: Multi-view acquisition
FVV: (for Free Viewpoint Video) Live viewpoint selection.
Problems Acquisition: Synchronization, calibration. . . Compression: Compact representation of the huge amount of data. Rendering: Photo-realistic virtual view generation. Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
Figure: 3D rendering April 1, 2010
2 / 19
Table on contents
1
Introduction
2
Incremental-LDI construction scheme
3
Ghosting
4
Results
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
3 / 19
Outline
1
Introduction
2
Incremental-LDI construction scheme
3
Ghosting
4
Results
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
4 / 19
View generation (Warping algorithm) Depth i
View i
Image-based rendering Input: View and associated Depth map Output: New viewpoint (texture & depth).
Warping
Problems Sampling: Visual artifacts ⇒ Inpainting Disocclusion: Unknown texture. ⇒ Extra information (LDI).
Figure: Warping algorithm Vincent JANTET (IRISA - France)
Figure: Disocclusion I-LDI for Multi-View Coding
April 1, 2010
5 / 19
LDI (Layered Depth Image) - [SGHS98, YLKH07]
A Layered Depth Image is a set of many layers constituted by depth pixels. Contains texture of occluded area.
··· 1st layer
2nd layer
3rd layer
4th layer
Figure: First layers of an LDI frame.
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
6 / 19
Classical LDI construction [CSY07] Merging policy
Every input views are warped onto a reference viewpoint, and then merged together.
View i
Eliminate pixels described twice.
Reference viewpoint
Merging
.
LDI
View j
.. .
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
7 / 19
Advantages and limits Advantages Disocclusion: Filled by real texture. Camera freedom: Virtual camera can move inside a large area. Compactness: Eliminate some correlated pixels and reduce data size.
Limits Compression: Many layers, partially empty with scattered pixels distribution. Visual artifacts: Ghosting, Bluring, . . .
Figure: Scattered pixels distribution. Vincent JANTET (IRISA - France)
Figure: Ghosting artifacts.
I-LDI for Multi-View Coding
April 1, 2010
8 / 19
Outline
1
Introduction
2
Incremental-LDI construction scheme
3
Ghosting
4
Results
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
9 / 19
Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view.
Viewpoint i
This information is warped back into the I-LDI.
Vincent JANTET (IRISA - France)
Warp Ref to i
I-LDI for Multi-View Coding
.
I-LDI
April 1, 2010
10 / 19
Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view.
Viewpoint i
View i
This information is warped back into the I-LDI.
Vincent JANTET (IRISA - France)
Exclusion difference
Warp i to Ref
Warp Ref to i
I-LDI for Multi-View Coding
.
I-LDI
April 1, 2010
10 / 19
Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view.
Viewpoint i
View i
This information is warped back into the I-LDI.
Vincent JANTET (IRISA - France)
Exclusion difference
Warp i to Ref
Insert Warp Ref to i
I-LDI for Multi-View Coding
.
I-LDI
April 1, 2010
10 / 19
Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view.
Viewpoint j
View j
This information is warped back into the I-LDI.
Vincent JANTET (IRISA - France)
Exclusion difference
Warp j to Ref
Insert Warp Ref to j
I-LDI for Multi-View Coding
.
I-LDI
April 1, 2010
10 / 19
Outline
1
Introduction
2
Incremental-LDI construction scheme
3
Ghosting
4
Results
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
11 / 19
Ghosting Artifacts Ghosting is due to pixels with blended color between background and foreground. Detect depth discontinuity [Canny]. Classify background and foreground pixels near each boundaries. Ignore background blended pixels from data.
First layer of an I-LDI frame.
Depth Map
Canny edges
Vincent JANTET (IRISA - France)
Boundaries
Ghost removal
I-LDI for Multi-View Coding
Figure: Ghosting artifacts removal April 1, 2010
12 / 19
Outline
1
Introduction
2
Incremental-LDI construction scheme
3
Ghosting
4
Results
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
13 / 19
First layers of an LDI and an I-LDI Comparison. LDI frame: many pixels, scattered pixels distribution, ...
... st
(a) 1 layer
nd
(b) 2
layer
rd
(c) 3
layer
th
(d) 4
layer
(e)
I-LDI frame: less layer and less pixels, compact distribution, ...
... st
(f) 1 layer
Vincent JANTET (IRISA - France)
nd
(g) 2
layer
rd
(h) 3
layer
I-LDI for Multi-View Coding
th
(i) 4
layer
April 1, 2010
(j)
14 / 19
LDI & I-LDI comparison.
(a) Layers completion rate.
Vincent JANTET (IRISA - France)
(b) Pixels ratio taken from different views.
I-LDI for Multi-View Coding
April 1, 2010
15 / 19
PSNR Original I-LDI
Original
SSIM Original I-LDI
Original 79.68%
LDI 79.62% 99.52%
Figure: PSNR & SSIM
Figure: LDI rendering.
(a) LDI.
(b) I-LDI.
Figure: Differences.
Figure: I-LDI rendering. Vincent JANTET (IRISA - France)
30.22
LDI 30.23 46.26
I-LDI for Multi-View Coding
April 1, 2010
16 / 19
Rendering result.
Figure: Rendering result of an I-LDI. Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
17 / 19
Rendering result.
Figure: Rendering result of an I-LDI. Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
17 / 19
Rendering result.
Figure: Rendering result of an I-LDI. Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
17 / 19
Conclusion
Advantages
Limits
80% less pixels for the same quality. Less correlation between layers. Compact pixels distribution.
Sampling artifacts. Some textures will never be inserted into the I-LDI.
Future works. Look for an efficient I-LDI compression algorithm. Questions?
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
18 / 19
References I
X. Cheng, L. Sun, and S. Yang. Generation of layered depth images from multi-view video. Image Processing, 2007. ICIP 2007. IEEE International Conference on, 5:V –225–V –228, 16 2007-Oct. 19 2007. J. Shade, S. Gortler, L. He, and R. Szeliski. Abstract layered depth images. 1998. S.-U. Yoon, E.-K. Lee, S.-Y. Kim, and Y.-S. Ho. A framework for representation and processing of multi-view video using the concept of layered depth image. Journal of VLSI Signal Processing Systems for Signal Image and Video Technology, 46:87–102, 2007.
Vincent JANTET (IRISA - France)
I-LDI for Multi-View Coding
April 1, 2010
19 / 19
