Real-time Action Selection for ECA Listeners Etienne de Sevin, Elisabetta Bevacqua and Catherine Pelachaud CNRS - Telecom ParisTech 37/39, rue Dareau 75014 Paris, France {de-sevin, bevacqua, pelachaud}@telecom-paristech.fr
Abstract We aim to build a real-time Embodied Conversational Agent able to generate automatically verbal and non verbal backchannels that a human interlocutor displays during an interaction. In this paper, we propose a backchannel selection algorithm working in real-time to choose the more appropriate backchannels to display among the possible conflicting ones according to the level of user’s interest perceived by the agent. Keywords: Backchannels, User’s interest level, Personality, Action selection, ECAs
3. Action selection algorithm Agent’s Mental State
Triggering of backchannels
Perceived User’s Level of Interest
Reactive Backchannels Backchannel Selection Mimicry Backchannels
User
ECA
Fig. 1. Schematic view of the Backchannel architecture including a BC selection module
1. Introduction To be believable, virtual agents have to deal with the problem of action selection which can be resumed to decide what to do next according to the internal and external variables of the agent [1]. This work is part of the EU SEMAINE project in which a real-time Embodied Conversational Agent (ECA) will be a Sensitive Artificial Listener (SAL) [2]. The SEMAINE project partners [3] provide us the interest level of the user by interpreting the audio (microphone). We implement an action selection to listener ECA in order to choose the right backchannel [4] to display according to the user behaviors. In this paper, we want to study the influence of the interest level of the user on the selection of backchannels.
Backchannels can be potentially conflicting because only one signal can be displayed on one modality of the agent (i.e. head). To resolve this conflict, we use the interest level of the user. We integrate the value of the user interest level provided by the SEMAINE partner [3] into a temporal window in order to neglect the non significant variations. The selection algorithm has to choose between two types of backchannels (see figure 1). Mimicry is chosen preferentially when the agent perceives that the user is very interested in the interaction so that the agent can show its high engagement in the interaction [5]. When fully engaged in an interaction, mimicry of behaviors may happen [6].
The reactive backchannels are used when the ECA detects that the user looses interest in the interaction and to encourage the user to be interested in the interaction [7]. As the action selection algorithm receives all the potential backchannels with their priorities from the action proposers (mimicry and response backchannel modules, see figure 1), the algorithm proceeds as follows: • Calculation of the priorities of BCs according to the user’s level of interest (estimated by the agent) • Selection of the most appropriate backchannels among possible conflicting ones to be sent to be display by the ECA. • Wait until the chosen backchannel is finished to be displayed by the player before choosing another one. Incoming Backchannels are queued and used during the next selection pass.. All backchannel priorities are normalized according to level of user’s interest in order to be able to compare them and make a selection. The selection is event-based and is done in real-time.
4. Conclusion and future work In this paper, we have presented our selection algorithm for backchannel signals. Depending on the user’s interest level, it chooses if the agent should mimic a user’s behaviour or should display a reactive or a responsive backchannel in order to have a better adaptation of the ECA to the user. In the future, we plan to integrate some internal variables of the ECA in the choice of backchannels such as personality and emotions [8]. Moreover, as our action selection algorithm is generic, it can also be used in an application to select actions in a gaze-based sharing attention context [9].
Acknowledgements The research leading to these results has received funding from the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement No. 211486 (SEMAINE).
References [1] E. de Sevin and D. Thalmann. A motivational Model of Action Selection for Virtual Humans. In SocietyPress, I., ed. : Computer Graphics International (CGI), pp.213-220, 2005. [2] E. Douglas-Cowie, R. Cowie, C. Cox, N. Amir and D. Heylen. The Sensitive Artificial Listener: an induction technique for generating emotionally coloured conversation. In : LREC2008, 2008. [3] B.Schuller et al. Being Bored? Recognising Natural Interest by Extensive Audiovisual Integration for Real-Life Application. Image and Vision Computing Journal. Special Issue on Visual and Multimodal Analysis of Human Spontaneous Behavior. ELSEVIER, Vol. 27, Issue 12, pp. 17601774, 2009. [4] E. Bevacqua, M. Mancini and C. Pelachaud. A Listening Agent Exhibiting Variable Behaviour. In Prendinger, Helmut, Lester, James, Ishizuka, Mitsuru, eds. : Proceedings of 8th International Conference on Intelligent Virtual Agents, Tokyo, Japan, vol. 5208, pp.262-269, 2008. [5] J. Gratch, A. Okhmatovskaia, F. Lamothe, S. Marsella, M. Morales and R. van der Werf. Virtual Rapport. 2006. [6] J. Lakin, V. Jefferis, C. Cheng and T. Chartrand. Chameleon Effect as Social Glue: Evidence for the Evolutionary Significance of Nonconsious Mimicry. Nonverbal Behavior 27, 145-162, 2003. [7] C. Goodwin. Conversational Organization Interaction between Speakers and Hearers. Academic Press. 1981. [8] C. Peters, S. Asteriadis, K. Karpouzis, E. de Sevin. Towards a Real-time Gazebased Shared Attention for a Virtual Agent. In : Workshop on Affective Interaction in Natural Environments (AFFINE) in the Tenth International Conference on Multimodal Interfaces (ICMI’08), 2008. [9] E. Bevacqua, E. de Sevin, C. Pelachaud, M. McRorie and I. Sneddon: Building Credible Agents: Behaviour Influenced by Personality and Emotional Traits. In Proceedings of International Conference on Kansei Engineering and Emotion Research 2010, KEER 2010. Paris, France, 2010.
