Object-based Layered Depth Images for improved virtual view synthesis in rate-constrained context Vincent Jantet 1 , Christine Guillemot 2 , Luce Morin
3
1
ENS Cachan, Antenne de Bretagne, Campus de Ker Lann, 35170 Bruz – France INRIA Rennes, Bretagne Atlantique, Campus de Beaulieu, 35042 Rennes – France 3 IETR - INSA Rennes, 20 avenue des Buttes de Coësmes, 35043 Rennes – France
2
ICIP 2011
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
1 / 21
Context of multi-view videos
Functionalities: 3DTV: Depth feeling by stereo-vision simulation. FVV: Live viewpoint selection. Require a virtual view synthesis method. Fig: 3D rendering
Acquisition
Compression
Rendering
Representation
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
2 / 21
Table on contents
1
Introduction
2
Object-based classification
3
Rendering results
4
Compression results
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
3 / 21
Outline
1
Introduction
2
Object-based classification
3
Rendering results
4
Compression results
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
4 / 21
Depth Image-Based Rendering (DIBR) Warping algorithm View i
Depth i
Input: View and associated depth map Output: New viewpoint (texture & depth)
Disocclusions Warping
Vincent Jantet (ENS-Cachan – France)
Obstructed scene information from reference viewpoint They appear along depth discontinuities Solution: Add additional informations (LDI)
Object-based LDI
Fig: Disocclusion
ICIP 2011
5 / 21
Layered Depth Image (LDI) [SGHS98] A set of layers, containing depth pixels from a single viewpoint
From a reference viewpoint 1st layer ⇒ Reference view Visible texture and depth 2nd layer ⇒ Residual data Hidden texture and depth
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
6 / 21
Layered Depth Image (LDI) [SGHS98] A set of layers, containing depth pixels from a single viewpoint
From a reference viewpoint 1st layer ⇒ Reference view Visible texture and depth 2nd layer ⇒ Residual data Hidden texture and depth
1st layer
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
6 / 21
Layered Depth Image (LDI) [SGHS98] A set of layers, containing depth pixels from a single viewpoint
From a reference viewpoint 1st layer ⇒ Reference view Visible texture and depth 2nd layer ⇒ Residual data Hidden texture and depth
1st layer
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
2nd layer
ICIP 2011
6 / 21
LDI from real multi-view plus depth sequence [JMG09]
Compressed depth map 1st
layer
2nd
layer
Limitations - Redundant boundaries in both layers - Moving elements in both layers - Layers contain large depth discontinuities (Discontinuities are hard to compress)
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
Synthesized virtual view Fig: Rendering impact of depth map compression ICIP 2011
7 / 21
Outline
1
Introduction
2
Object-based classification
3
Rendering results
4
Compression results
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
8 / 21
Object-based LDI representation This representation organizes pixels into layers to enhance depth continuity
1st layer
2nd layer
Fig: Classical LDI depth layers
Foreground
Background
Fig: Object-based LDI depth layers
Method based on a growing-region algorithm Region R initialized with pixels where ZFG and ZBG are already defined. For each pixel q outside R: - Extrapolate ZFG and ZBG . - Classify q.
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
9 / 21
Object-based LDI representation This representation organizes pixels into layers to enhance depth continuity
1st layer
2nd layer
Fig: Classical LDI depth layers
Foreground
Background
Fig: Object-based LDI depth layers
Method based on a growing-region algorithm Region R initialized with pixels where ZFG and ZBG are already defined. For each pixel q outside R: - Extrapolate ZFG and ZBG . - Classify q.
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
9 / 21
Classification: Initializing
Foreground Vincent Jantet (ENS-Cachan – France)
Unclassified Object-based LDI
Background ICIP 2011
10 / 21
Classification: Results
Foreground Vincent Jantet (ENS-Cachan – France)
Unclassified Object-based LDI
Background ICIP 2011
11 / 21
Background inpainting [CPT03] Advantages - Remove unnecessary boundaries - Inpainting processed once, before encoding stage - No need of inpainting method at rendering stage
Fig: Background inpainting
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
12 / 21
Outline
1
Introduction
2
Object-based classification
3
Rendering results
4
Compression results
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
13 / 21
Rendering results
Classical LDI rendering Virtual view inpainting O-LDI rendering
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
14 / 21
Fast mesh-based rendering
Fig: Object-based LDI
Continuous layers can be rendered as meshes. Foreground mesh is partially transparent. Vincent Jantet (ENS-Cachan – France)
Object-based LDI
Fig: Meshes rendering
ICIP 2011
15 / 21
Outline
1
Introduction
2
Object-based classification
3
Rendering results
4
Compression results
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
16 / 21
LDI and MVD compression schemes MVD compression (MVC) LDI compression (MVC)
Input views
V1
V3
V5
LDI4 Input LDI
V7
MVC
MVC V10
V30
V50
LDI40 Compressed LDI
V70
Compressed views
Rendering
Compression
DIBR
Rendering
VSRS V6 Final view
Final view
Vincent Jantet (ENS-Cachan – France)
V600 Object-based LDI
ICIP 2011
17 / 21
Rate-distortion curve
SSIM (%)
90
88
86
Object-based LDI Classical LDI —– MPEG (MVC/VSRS)
84 0 Vincent Jantet (ENS-Cachan – France)
5
10 15 Bitrate (Mbit/s) Object-based LDI
20 ICIP 2011
18 / 21
Conclusions on Object-Based LDI
Advantages -
Remove unnecessary boundaries ⇒ Improve compression Static background along time Compatible with fast mesh-based rendering Depth continuity improves rendering quality
Limit - No backward compatibility with 2D decoding scheme Questions?
Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
19 / 21
[CPT03]
A. Criminisi, P. Pérez, and K. Toyama. Object removal by exemplar-based inpainting. In Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society Conference on, volume 2, pages 721–728, June 2003.
[JMG09] Vincent Jantet, Luce Morin, and Christine Guillemot. Incremental-ldi for multi-view coding. In 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, pages 1–4, Potsdam, Germany, May 2009. [SGHS98] Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In SIGGRAPH ’98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 231–242, New York, NY, USA, July 1998. ACM. Vincent Jantet (ENS-Cachan – France)
Object-based LDI
ICIP 2011
20 / 21
