Stereoscopic 3-D video for the human eyes Frédéric Devernay, INRIA Grenoble - Rhône-Alpes research done within the 3DLive project with Sergi Pujades, Elise Mansilla, Loïc Lefort, Martin Guillon, Matthieu Volat, Sylvain Duchêne Images 3D : acquisition, synthèse et visualisation October 114, 2010
Bright objects appear to beperception nearer than dimbeones, and an in o may found Light and look shade. colors like they’re closer than dark ones. Interposition is so obviousNot it’s Relative size. Monocular Cues Brig moti handbook you are now looking Images which are rich in th Fr´ed´eric Devernay and Paul Beardsley Relative size involves the size of thesay image of an obj spati colo The monocular, or extrastereoscopic, depth cues are the basis fo is visualize behind it, your desk, beca when the binocul perception depth in visual displays,We and are justthat as important lens of theofeye onto the retina. know objects interposed between you andindic ob stereopsis for creating which are they perceived as trulytwo three Rel they are closer, andimages smaller when are farther awa Light and shade provide dimensional. These cues include light and shade, relative size, a dept lens Textural gradient is the only m us to make a judgment about the distance of familiar interposition, aerial perspective, motion parall Textural gradient.textural gradient,objects look solid or round isthey m psychologist in modern times. seen at some great distance is interpreted to be far aw most importantly, perspective. an A more complete description of object appear to be man resti us t
Three-Dimensional Depth Cues
8
Interposition. LightLight and shade and shade.
painters byonthe time of psycholo the Rena perception may be found in a basic text perceptual Relative size Interposition Interposition. of an Relative size. seen lawn or the tweed a jacket, p Interposition is so obvious it’s taken for of granted. Yo
Bright objects appear bec stere apparent the object is Images which are rich in the more monocular depthascues will beto even handbook you are now looking at is closer to you or i colors look they’recue. clo visualize when the binocular stereoscopic cuelike is added. Inte
is behind it, say your desk, because you can’t see thro Aerial perspective is the dimin han interposed between you and depth objects which are farther Light and shade provide a basic cue. Artists learn how Pers caused by intervening haze. isOf bt Relative size involves the objects look solid or rounded by shading them. Cast shadows c “pho haze because of theonto scattering o inte lens of the eye the ret an object appear to be resting on a surface. Textural gradient is the only monocular depth cue a Textural gradient Aerial perspective Perspective Perspective. In thick fog or haze, objects ma Aerial perspective. dept Textural gradient. they are closer, and smalle Relative size. psychologist in modern times. The other cues were k Com Tex Fig. 1 Six monoscopic depth cues (from [60]). seventh is motion parallax, which hard BrightThe objects appear to beTextural nearer dim ones, andtoobjects w us than togradient. make ais judgment abo painters thethey’re time of the Renaissance. m illustrate, and depth of field can also be considered asby alike depth cue (see Fig.than 3). dark ones. A textured and psy colors look closer Interposition. seen at some great distance lawn or the tweed of a jacket, provides a depth cue be ispain th 3 of is Relative size involves theobject size thecloser image of project more apparent as the to an theobject observer. exag law is soappear obviou lens of the eye onto the retina. Interposition We know that objects lar vani mor distance of 3m, a depth of field of ±they 0.3D, meansandthat the when in-focus is from are closer, smaller they range are farther away. Memo handbook you are now lood Aerial perspective is the diminution in visibility of 1 1 2 1/( + 0.3) ≈ 1.6m to 1/( − 0.3) = at about a focus distanceofof 30cm, us 30m, to makewhereas a judgment the distance familiar objects. A
And also motion parallax, depth of stereoscopy field, and... stereoscopy
Depth of field as a depth cue: focus matters! 3
Conflicting depth cues • The 9 cues may give opposite indications on the scene geometry
•
The pseudoscope (Wheatstone) - reverse left and right eyes - causes closer objects to seem even bigger:
•
big in the image
•
binocular disparity indicates they are also far away
4
William Hogarth, 1754
Conflicting cues: Ames room
Used in Lord of the Rings, Eternal Sunshine of the Spotless Mind... 5
Stereoscopic conflicting cues: Coraline 3D
Coraline (H. Selick & P. Kozachik)
2 vanishing points in the same 3-D scene 6
Stereoscopic conflicting cues: Coraline 3D
Coraline (H. Selick & P. Kozachik)
2 vanishing points in the same 3-D scene 6
Stereoscopic conflicting cues: Coraline 3D
Coraline (H. Selick & P. Kozachik)
2 vanishing points in the same 3-D scene 6
Stereoscopic conflicting cues: Coraline 3D
Coraline (H. Selick & P. Kozachik)
2 vanishing points in the same 3-D scene 6
Stereo-specific processes
7
Stereo-specific processes • Correcting causes of visual fatigue
7
Stereo-specific processes • Correcting causes of visual fatigue • Adapt the movie to the screen size
7
Stereo-specific processes • Correcting causes of visual fatigue • Adapt the movie to the screen size • Global 3-D changes (interocular, infinity...)
7
Stereo-specific processes • Correcting causes of visual fatigue • Adapt the movie to the screen size • Global 3-D changes (interocular, infinity...) • Local 3-D changes (3-D touchup)
7
Stereo-specific processes • Correcting causes of visual fatigue • Adapt the movie to the screen size • Global 3-D changes (interocular, infinity...) • Local 3-D changes (3-D touchup) • Playing with the depth of focus 7
Stereo-specific processes • Correcting causes of visual fatigue • Adapt the movie to the screen size • Global 3-D changes (interocular, infinity...) • Local 3-D changes (3-D touchup) • Playing with the depth of focus • Playing with the proscenium 7
Stereo-specific processes • Correcting causes of visual fatigue • Adapt the movie to the screen size • Global 3-D changes (interocular, infinity...) • Local 3-D changes (3-D touchup) • Playing with the depth of focus • Playing with the proscenium • 3-D compositing (real or CG scenes) 7
A few definitions • Two cameras, two eyes • Screen plane ... in the viewer space • Plane of convergence .. in the scene space • 3-D cone • Interocular / Interaxial • bigger than 65mm (can be 30m)⟹ hyperstereo (or miniaturization)
• smaller than 65mm (can be 0cm) ⟹ hypostereo (or gigantism)
• Convergence 8
Binocular disparity: why we see in 3D
• Objects at different depths cause different disparities Disparity
9
left view
10
right view
11
The proscenium arch (or stereoscopic window) The screen is a window on the world If object closer than convergence plane touches the image borders... ⟹ Add black borders to move proscenium arch closer
12
Visual fatigue (1) a critical point • Can lead to: • a simple headache • temporary or permanent damage to the
oculo-motor system (especially on children)
• A public health problem (just as the
critical fusion frequency on CRT screens...)
13
14
Visual fatigue (2) geometric differences
Fr´ed´eric Devernay and Paul Beardsley
Fig. 6 A few examples of geometric asymmetries: (a) Vertical shift, (b) Size or magnification difference, (c) Distortion difference, (d) Keystone distortion due to toed-in cameras, (e) Horizontal shift - leading to eye divergence in this case (adapted from Ukai and Howarth [66]).
a. vertical shift b. size difference c. distortion difference should be avoided”. But they also went on to say, in listing future development requirements, that “Much experimental work must be carried out to determine limitd. keystone (toed-in cameras) ing values of divergence at different viewing distances which are acceptable without e. horizontal shift (divergence...) eyestrain”. These limiting values are the maximum disparities acceptable around the convergence point, usually expressed as angular values, such that the binocular fu14
Visual fatigue (3) accommodation and convergence discrepancy
EMOTO et al.: REPEATED VERGENCE ADAPTATION CAUSES THE DECLINE OF VISUAL FUNCTIONS IN WATCHING STEREOSCOPIC TV
distance of accommodation = distance to screen ≠ distance of convergence
(a)
Different display Different depth of field:
• 3DTV (3.5m): 2m →12m • Movie theater (16m): 4m → infinity
(b)
Emoto et al. 2005
issues involving hardware (leading to differences between views of left and right TV images). The factor involving the principle of stereoscopic TV should be investigated first. Binocular parallax can be controlled during the recording of stereoscopic images, and it is therefore a problem of software production. Hardware factors, outside the scope of our current investigation, have been discussed in many published papers [29]–[39]. In most of those studies, visual comfort for shortterm viewing was assessed, but visual fatigue from long-term viewing was not discussed directly, though it does have impact on visual comfort. Even if the hardware difference is eliminated, control of binocular parallax load is still difficult. It is not possible to pre-determine what object will be viewed by the viewer, or the level of binocular parallax that viewed object may have while recording the video. In some studies, the maximum amount of binocular parallax is described [24], [26]. It is difficult to know the amount of binocular parallax load viewers experience in experiments, because it is necessary to control the image viewing position, to determine where the viewers see, and calculate the amount of binocular parallax by stereo-matching 15 [40]. Despite this difficulty, it is essential to control the amount
Visual fatigue (4) screen size One 3-D movie, different screens ⟹ risk of divergence Shifting the images solves divergence problems, but creates other problems:
• Breaks the stereoscopic window • Causes depth distortions
109
Ukai & Howarth 2008
K. Ukai, P.A. Howarth / Displays 29 (2008) 106–116
16 system. (a) Far objects should have separation equivalent to IPD. Fig. 2. Method for avoiding diverged binocular visual axis, assuming double projection (b) However, usually it is difficult to know actual screen size when taking a movie, so that sometimes unexpected effect such as diverged binocular
Correcting geometric differences: the problem • Mechanics and optics are intrinsically imprecise • Check that the 3D movie can be comfortably viewed on a given screen (movie theater or 3DTV)
• On output, disparity must be purely horizontal • Transform the images to remove geometric differences
17
DisparityTakker: The Binocle / INRIA solution
• Detect remarkable points or regions in both images
• Match these points and regions • Compute image transformations to remove vertical disparities
• Real-time correction of HD-SDI
stereoscopic streams (2 x 1080p60) 18
19
20
21
22
23
Alerts for a 4m wide screen 24
Alerts for a 10m wide screen: crowd too close! 25
Alerts for a 10m wide screen + shift: divergence! 26
Global depth modifications: adapting to the display b H
camera
display
camera interocular
eye interocular
W
convergence screen distance distance
width of W convergence plane
dW H
screen size
Z
real depth
perceived depth
d
disparity (as a fraction of W) 27
b
Z
12.5cm 10
hyperstereo
6.5cm
5
hypostereo
real depth -10
-5
0.5cm 0 -5 -10 -15
perceived depth
-15
Global depth modifications: changing b (camera interocular)
5
10
28
15
20
25
30
35
10
12.5cm 6.5cm
hyperstereo
hypostereo
5
real depth -10
-5
0.5cm 0 -5 -10
5
10
15
20
25
30
z’=z close to the screen perceived
-15
Global depth modifications: H = αb
Keep object image size constant: object farther/closer, zoom in/out, interocular proportional to distance 29
35
Perceived depth distortions
•
• • •
3D geometry is not distorted if and only if shooting and viewing geometry are the same
• • •
used for IMAX-3D impossible in real situations (sports...) may break the stereoscopic window
Objects don’t look «more 3D» on a bigger screen Distance is important: «more 3D» if screen farther New view synthesis is the only solution (requires depth map) 30
New view synthesis: baseline modification
Scene geometry
Viewing geometry
Objects on screen are not distorted, but everything else is very distorted! Divergence may happen! 31
New view synthesis: viewpoint modification
Scene geometry
Viewing geometry
No distortion at all, but many objects cannot be seen in the original images... bad solution! 32
New view synthesis: disparity remapping
Scene geometry
Viewing geometry
Best tradeoff: depth is not distorted, no divergence happens, only apparent width is distorted... like on any 2D image 33
New view synthesis: how we do it
• Video-rate depth map computation • Computation done on the GPU • Will be included in Binocle
DisparityTakker in 2011 for the 3DLive project
• Can also be done in a set-top box on the display side (by Technicolor) 34
