How auditory information influences binocular rivalry: Revisiting the McGurk effect Manuel Vidal and Victor Barrès
Laboratoire de Physiologie de la Perception et de l’Action, Collège de France, Paris
[email protected]
Introduction Although multistable perception has long been studied, in recent years, paradigms involving ambiguous visual stimuli have been used to assess phenomenal awareness. Indeed, for such stimuli, although the sensory input is invariant, a perceptual decision occurs bringing to awareness only one of the possible interpretations. Moreover, the observer experiences inevitable oscillations between these percepts.
Only few studies have tackled bistability for multimodal percepts. Hupé et al. [1] used spatially located audio-visual beeps and failed in finding synchronized auditory and visual perceptual decisions. Van Ee et al. [2] investigated how sensory congruency could influence volitional control in binocular rivalry using looming motion stimuli that could be seen, heard or sensed on the hand. They found that attention to the additional modality
(sound or touch) has to be engaged to promote visual dominance and that only temporal congruency is required in the end. Nonetheless, we believe that in both studies such low-level stimuli could have reduced the possibility of finding passive interactions and limited the congruency to temporal features, which motivated the use of higher-level audio-visual verbal processing in our experiments. In a clever dynamic version of the classical face/vase
illusion, Munhall et al. [3] found no interaction between the suppressed percept (the talking faces) and sound. Based on known differences between ambiguous figures and rivalry, we wondered if the use of rivaling talking faces could increase the sensory integrations with the suppressed percept. We conducted a series of experiments to study multimodal rivalry based on the McGurk effect [4], known to involve robust audio-visual integration.
Methods and baseline experiments · 14 naïve subjects
· Control of correct BR performance with static lips: – Low piecemeal: 6.83±1.50% – Gamma distribution of dominance durations – Black Lips stimulus stronger than White Lips [5] which interacts with the dominant eye [6] · RT measured with forced stimulus flip: 425±18ms
· Visual input: dual channel Head Mounted Display (nVisor SX at 1280x768), 5º circular aperture · Auditory input: ear canal phones (Sennheiser)
· Visual percept dominance durations (continuous) · Audio-Visual percept (single) with forced choice
Left eye
Right eye
=
/aba/
/aba/
Left eye
=
/aga/
or
/aga/
O
1.85s
McGurk group No McGurk group
100% 90% 80% 70% 60% 50% 40% 30% 20%
/aba/
0% /aga/ seen
Visual stimulus (/aba/ heard)
Left eye
/aga/
Esc
/aga/
O
+tS
+1.85s
Space
+RT
/aba/
Wait for target percept
/ /ada/
/aba/
Left eye
Hold
/aba/
/
130s
Right eye
Instructions Esc
O
/aba/
Loop duration: 1.65s
Instructions
Left eye
Exp. 5 − Continuous visual report for video rivalry with sound
Right eye
/aba/
Right eye
/
/ /ada/
Hold
Left eye
Wait for
Exp. 3 − Continuous visual report for video rivalry
Large proportion of people not sensitive to the McGurk effect, let’s use them!
10% /aba/ seen
– McGurk group: n=8 all with more than 90% of AV/ada/ heard – No McGurk group: n=6 all with less than 10% of AV/ada/ heard
/aba/
+RT
{
McGurk effect
Right eye
· Test of the McGurk effect (A/aba/ + V/aga/ → AV/ada/):
Selected /aba/ percept
Data analyzed
/aba/
Experiment 2
Protocols · Task: continuous report during 130s exposure for dominance duration studies (exp. 1, 3 & 5) or single report after each audio-visual stimuli (exp. 2 & 4) · Volition test: hold specific stimuli (continous) · Lips motion x color balanced · 3 (continuous) or 20 (single) repetitions
Exp. 4 − Single AV report with video rivalry (visual dominant)
{ {
Material
Exp. 2 − Single AV report (McGurk effect baseline)
{ {
Experiment 1
O
130s
Esc
Only when volition tested
/aga/
Loop duration: 1.65s
Instructions
Right eye
{
Participants
Only when volition tested
Results and discussion
8
90% 80% 70% 60% 40% 30%
Duration (s)
3 2 1 0 Black lips
White lips
Black lips
McGurk
White lips
No McGurk
These findings show that our lips motion videos share the common characteristics of binocular competition (exclusivity, randomness, inevitability), providing a validation of this innovative tool to study high-level audiovisual integration in rivalry.
Experiment 4 (McGurks group only) · Audio-visual fusion with lips motion in suppressed eye (Fig. B) when compared to the baseline without rivalry (exp. 2): · Not different than noise level when lips are seen is in dominant eye
/aba/ seen
/aga/ seen
/aba/ seen
McGurk group
· For the McGurks (Fig. E, left): the addition of A/aba/ increased the V/aba/ relative dominance when holding V/aba/ (due to a decrease of V/aga/ duration [19]) but not when holding V/aga/
/aga/ seen
No McGurk group
Visual stimulus
10% /aba/ seen
/aga/ seen
/aba/ seen
/aga/ seen
in dominant eye
/aba/ seen
/aga/ seen
in non-dominant eye
These results suggest that the suppressed percept is not deactivated and can be available for further processing, which in turn can rise up to awareness. Indeed, when the hidden stimulus is presented in the dominant eye a non-negligible proportion of integration with the auditory input seems to occur. This was made possible with the natural talking faces stimuli presented in video rivalry instead of the face/vase illusion used in [3].
Contrarily to [2] conclusions, our results show that with high-level audio-visual stimulation presenting robust and natural integration, passive viewing can in fact increase the dominance durations for the McGurk subjects (those merging lips with sound). · For the McGurks (Fig. D): in the hold V/aba/ condition there is differential effect of the consistent sound A/aba/ on the V/aba/ relative dominance [16]: – Without sound: increase of 1.00s [17] – With A/aba/: increase of 2.70s [18] – This was not the case for the No McGurks D. Differential effects of sound on volition
Experiment 5
3.5
· Passive viewing (Fig. C): when adding A/aba/ (no sound baseline from exp. 3) there is an interaction effect with the group on durations [13]: – McGurks show a dominance duration increase of 1.14s [14] – No McGurks show a marginal dominance duration decrease of 0.88s [15] – Independently of the lips movement (/aba/ or /aga/) for both groups · Switch distribution: no probability increase during lips motion
3. Munhall, K., ten Hove, M., Brammer, M. & Paré, M. Audiovisual Integration of Speech in a Bistable Illusion. Current Biology 19, 735-739 (2009) 4. McGurk, H. & MacDonald, J. Hearing lips and seeing voices. Nature 264, 746-748 (1976)
F(1,12) = 13.03, p
