Accurate On-line Avatar Control with Collision Anticipation Damien Maupu, Daniel Raunhardt, Daniel Thalmann, Ronan Boulic

Manuel Peinado1, Daniel Meziat Departamento de Automática. Universidad de Alcalá, Spain

VRLab. Ecole Polytechnique Fédérale de Lausanne, Switzerland

However, if that assumption does not hold, these systems produce postures in which the avatar penetrates its environment. This phenomenon is well-known in realistic virtual grasping and is usually referred to as “visual sink-in” [Borst and Indugula 2005]. Since visual artifacts such as penetrations are known to be more distressing for the user than introducing mapping discrepancies, most systems deal with them by breaking the direct mapping of motion from the participant to the avatar. A common approach for doing so is the rubber band method, which is conceptually illustrated in Figure 1a.

Abstract Interactive control of a virtual character through full body movement has a wide range of applications. However, there is a need for systems that accurately reproduce the motion of a performer while accounting for surrounding obstacles. We propose an approach based on a Prioritized Inverse Kinematics constraint solver. Several markers are placed on the user’s body. A set of kinematic constraints make the virtual character track these markers. At the same time, we monitor the instantaneous displacements of a set of geometric primitives, called observers, attached to different parts of the virtual character. When an observer enters the influence area of an obstacle, its motion is damped by means of automatically created preventive constraints. The IK solver satisfies both maker and preventive constraints simultaneously, yielding postures of the virtual character that remain close to those of the user, while avoiding collisions with the virtual environment. Our performance measurements show the maturity of the IK technology for real-time full-body interactions.

Our system gets inspiration from the rubber band method, extending it in two ways. First, it deals with full-body motions, in contrast with traditional applications that permit manipulation tasks only. Second, our system extends the rubber band method with the notion of damping, i.e. the influence area surrounding the obstacle is becoming increasingly viscous as a body part is getting closer to the obstacle. With damping, instead of reacting after a collision has taken place, the system anticipates them by correcting the avatar’s motions as soon as the body part enters the damping region. This is illustrated in Figure 1b.

Categories and Subject Descriptors: I.3.6 [Computer Graphics]: Methodology and Techniques— Interaction techniques; I.3.7 [Computer Graphics]: ThreeDimensional Graphics and Realism—Virtual Reality Keywords: Virtual Reality, Motion Capture, Animation, Inverse Kinematics, Collision Avoidance

1

Character

Introduction

Figure 1: a) In the rubber band method, the avatar’s body part (white dot) remains tangent to the obstacle surface while the real body part collides. b) In our damping method, whenever an observer enters the influence area of an obstacle (frame 2), its displacement is progressively damped (2-3) to ensure that no interpenetration happens (4-6). No damping is exerted when moving away from the obstacle (7).

Interactive control of a virtual character, or avatar, through full body movement has applications in fields like virtual prototyping, workspace design, and computer animated puppetry. It is also a powerful and intuitive control paradigm when interacting with general purpose virtual environments (VEs). Commercial systems are available that accurately capture human motion under the assumption that the avatar is not surrounded by obstacles. 1

To meet the demands of full-body avatar motion control and of collision avoidance, our system employs a novel Prioritized Inverse Kinematics solver. This solver works by solving simultaneously constraints of two kinds. First, marker constraints are used to make the avatar follow the position of the input signals coming from the markers. The linearization performed by our IK technique corrects large errors over a few time steps, which is sufficient for general purpose full-body movements.

[email protected]

Copyright © 2007 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail [email protected]. VRST 2007, Newport Beach, California, November 5–7, 2007. © 2007 ACM 978-1-59593-863-3/07/0011 $5.00

Second, the set of marker constraints is extended by the collision avoidance system with dynamically created damping constraints.

89

recent research by Garber and Lin integrates repulsive constraints whose strength is a function of the distance to the obstacle that originates them [2002]. The problem with these approaches is that a body part is repelled from an obstacle even if it is not moving in its direction. We believe that our damping approach yields more natural results when applied to virtual humans.

For that purpose, a special entity, which we call an observer, is attached to each body part that must be checked for collisions (e.g. hand, elbow, head, etc...). An observer acts by creating a damping constraint to smoothly correct the motion of its controlled body part towards nearby obstacles (see Figure 1b, the arrow indicates the direction of the proposed displacement of the body part). Damping constraints are prioritized according to the risk of collision they pose.

Due to the computation complexity of human motion, randomized methods have been investigated to find collision free motions. They search in configuration space for a path linking a start and a goal configuration, and have been applied for the animation of virtual humans [Kallmann et al. 2007]. Since they cannot deal with full-body motions in real-time, their suitability for Virtual Reality applications is limited.

The integration of both families of constraints within the same prioritized IK scheme results in full body movements where collisions are mainly avoided by anticipation. Such avoidance is possible because the avatar skeleton is highly redundant, i.e. most of the time postures preventing collisions exist. With respect to the traditional approach of reacting only after collisions take place, we believe that our observer-based obstacle avoidance has a positive impact on the quality of the experience especially for the body parts that are out of the user’s attention focus, or out of his field of view (e.g. elbow, back of the head, etc…) We believe too that this results is a more accurate model of the way motion is executed, where collisions are dealt with in advance and obstacles are circumvented with a safety distance margin [Rosenbaum et al. 2001].

1.1

Also relevant to our work are those methods that specifically address dynamic environments. A system by van den Berg et al. computes plans in state-time space [2006]. Another common approach is to modify during execution time a pre-planned trajectory to cope with obstacles that move unexpectedly [Brock and Khatib 2002]. All these methods assume that some knowledge about the goal is available beforehand, so they are not applicable to our problem in which the participant’s movements are not known in advance. Finally, a very popular trend is to use the graphics hardware to accelerate collision tests [Govindaraju et al. 2006]. Kim et al. propose an efficient method to detect collisions between an avatar and its environment, but their response method is purely reactive [2007]. Exploiting the graphics hardware to accelerate collision tests could improve the performance of our method, without requiring major architectural changes.

Related Work

Full-body avatar control Interactive control of an avatar by direct transfer of body motions is not new. In fact, its appeal to fields such as live TV production or Virtual Reality has turned it into a widely studied subject of research [Sturman 1998]. Most systems require the use of extensive sensor or marker sets, which might render them too cumbersome or expensive for some applications. Less invasive solutions exist [Badler et al. 1993], often exploiting methods from the computer vision community, but they rarely enable accurate control of the full body.

Constrained virtual environments Many VR systems allow the participant to move in the VE using a direct mapping of his own body motion. This results in a natural, intuitive interface where the learning curve is very small. However, the freedom provided by this approach has a side-effect. Since the participant generally moves within the uncluttered space of a motion capture facility, nothing prevents him from moving in ways which are inconsistent in the VE. For example, he can stretch his arm forward even if his avatar is facing a near wall. Note that this is not the case if some sort of force-feedback is used, or if realobjects are included in the environment [Lok 2002], but here we focus only in systems where nothing constrains, in principle, a participant’s interaction with the VE.

A common approach to the motion transfer problem is to limit the range of allowed actions to a predefined, application-specific, set. The problem then becomes one of finding which action in the database best matches the current motion of the user [Lee et al. 2002]. These techniques lack the flexibility required for a generalpurpose posture reconstruction scenario. Besides, it is not clear how to extend them with collision avoidance capabilities. Of greater concern to the present work are methods that employ Inverse Kinematics. For performance reasons these methods usually divide the problem in different parts: first the body root is placed according to some optimization criteria, and then dedicated analytic IK solvers place the remaining body parts so as to solve the initial constraints [Badler et al. 1993; Tolani et al. 2000]. Our method employs a different approach to Inverse Kinematics, since user-defined sets of joints can be used for the satisfaction of each constraint, and since a different priority can be assigned to each constraint. This has been used in the past in specific contexts, like offline reconstruction of motions from a database where only a reduced subset of the input markers is known [Peinado et al. 2006]. In this work we extend this approach to online avatar control from a set of markers which is smaller (and thus less encumbering) than most production setups, yet at the same time allowing for an accurate reconstruction of postures.

Visual artifacts such as interpenetration greatly disrupt a participant’s sense of immersion. Therefore, research efforts have been conducted to provide ways of preventing them. We call the resulting systems “constrained virtual environments”, because the motion of the participant is constrained before being mapped to his avatar. Studies have demonstrated that, in terms of immersion, the amount of disruption produced by this indirect mapping (usually referred to as visual-proprioceptive discrepancy) is smaller than that caused by interpenetration [Burns et al. 2006a]. Thus the practical interest of such systems. Three methods have been proposed that differ in how this discrepancy is introduced: the rubber band method, the incremental motion method, and the credible avatar limb motion method [Burns and Brooks, 2006b]. The present approach is based in the rubber band method, extending it with the notion of collision anticipation thanks to the damping mechanism, and applying it to full-body control.

Collision avoidance in character animation Most approaches to collision avoidance in character animation are purely responsive; a collision is solved once it has occurred, but nothing is done before that instant [Zhao and Badler 1994]. Among the research efforts that try to anticipate collisions, the earliest one is probably due to Maciejewski and Klein who propose a two priority layer architecture [Maciejewski and Klein 1985]. A more

Whereas our system focuses on collision-free full-body avatar control, most previous efforts deal only with grasping and manipulation tasks. Such is the case of [Ullmann and Sauer 2000], where motion of the hands is corrected to avoid collisions during manipulation. Visual cues are provided to the user in the form of

90

IK iteration, this latter proposes an already damped posture. In most practical scenarios, however, the pre-IK target correction method alone does not suffice because there might be observers which do not have a marker constrain to act on. For example, Table 1 shows that in our posture reconstruction setting no positional marker constraint is used for the head of the avatar. Therefore, an observer attached to the head would have no opportunity to prevent collisions of that body part. Thus the need to perform another collision prevention stage after the main IK iteration.

“ghost hands”, a wireframe representation of where his actual hands would be had they not been corrected. Similar applications of this metaphor are present in the work of Borst and Indugula [2005]. All these works suggest that such visual cues allow a more efficient solution of complex tasks like those found in assembly simulation. However, in our system, where full-body motion can be corrected, showing a whole “ghost” body requests a significant amount of computing resource that we prefer to dedicate to the posture reconstruction process. A different approach to constrained motion is found in [Lindemann et al. 2001], where the participant’s posture is not corrected to avoid collisions, but to facilitate complex manipulation tasks, reducing their dimensionality. Another well-known method to correct the participant’s motion is called the Go-go technique [Bownman and Hodges 1997]. Here, the virtual hand moves outward faster than the physical hand, allowing the avatar to reach remote objects with ease. Finally, Lok presents a hybrid VE in which video captures of the participant are combined, and interact, with virtual objects [2002]. This allows for natural interactions, like drawing a curtain with the same arm movements that would be used on a physical one.

1.2

Another reason for performing post-IK collision prevention is that, in our IK solver, the increment computed for a constrained body part might be very different from the initial target of that constraint. This is due to the fact that positional constraints are satisfied according to a priority ordering, so one constraint with a given target might not be able to see this target satisfied because of a conflicting constraint of higher priority. The Posture Correction sub-layer (§5.1) performs the necessary post-IK prevention of future collisions. Each observer monitors the output of the IK solver, adding a preventive constraint to it when necessary. The goal of this preventive constraint is to damp the motion of the observed body part in the obstacle’s normal direction. Moving obstacles are supported too, thanks to a relativevelocity formulation in which the motion of an obstacle towards a body part is detected (and reacted to) as if the body part were moving and the obstacle remained still. This results in a reverse damping in which the observer dodges the approaching obstacle.

Method Overview

We present a method for the real-time full-body control of an articulated figure in an environment with obstacles. The method is applicable to figures with an arbitrary number of degrees of freedom, although we mainly focus on the control of virtual mannequins, or avatars.

As seen in Figure 2, if at least one observer has created a preventive constraint, a new IK iteration is performed and the resulting posture is used to update the proposed avatar pose. Multiple preventive constraints can be created at that stage. The IK re-evaluation should be repeated until no more damping is necessary, i.e. until no new preventive constraint is created. However, ensuring fluid interactions being a key requirement, we have set a maximum number of such additional re-evaluation of the IK solution. This may lead to small penetrations that this stage can also handle (§5.1).

The problem can be stated as follows: given an articulated figure, a description of its surrounding environment, and a set of input signals corresponding to the position of relevant points on the user body, update in real-time the avatar posture while avoiding collisions with nearby obstacles. Our system is independent of the specific methods by which these input signals are acquired (typical methods include optical and magnetic motion capture). For simplicity, we will henceforth assume that an optical method is used and refer to these signals as markers. One benefit of our method is that a reduced number of markers is needed compared to a production setup

While interacting with obstacles, the risk of collision regularly vanishes so no damping constraints remains and only marker constraints are solved by the IK solver. Thus, the avatar pose returns smoothly (thanks to the iterative nature of Prioritized IK) to the pose of the participant.

Given this problem statement, the solution we propose is summarized in Figure 2. At the core lies a state-of-the-art Prioritized IK solver (§2), which is able to satisfy a set of multiple Cartesian constraints according to a strict priority ordering [Baerlocher and Boulic 2004]. Working on an anatomically-correct human skeletal model (§3), it is able to produce realistic postures at real-time frame rates.

Start COLLISION RESOLUTION Yes

Yes

Collisions?

Create repulsive constraints

The online loop: from posture reconstruction to display

New obstacles?

POST-IK POSTURE CORRECTION Damped avatar pose

Yes

The Posture Reconstruction layer (§4) keeps a list of marker constraints. On each frame, it queries the position of the input markers and update the marker constraints accordingly. Whenever the IK solver satisfies these constraints, the full body motion of the participant is transferred to the avatar. However, this layer is unaware of the environment and may thus produce poses that penetrate surrounding obstacles.

IK iterations until collisions solved

Evaluate observer displacements

Accept solution Refresh display

Proposed avatar pose

POSTURE RECONSTRUCTION

Marker positions

New elements in CP? Updated preventive constraints CP

Collision-free avatar pose

Our Collision Prevention layer takes care of this. To do so, it keeps a list of observers. Each observer acts in two related ways. First, in the Target Correction sub-layer, only the observers of constrained body parts are allowed to act. A constrained body part is a body part whose motion is controlled by a marker, and thus has a positional marker constraint (see Table 1). The observer of a constrained body part evaluates the instantaneous target of the marker constraint, correcting it to damp its component in the obstacle’s normal direction. As this correction is done prior to the

Clear CP

IK Iteration CP +C’M Damped marker constraints C’M

Update marker constraints CM

PRE-IK TARGET CORRECTION Updated marker constraints CM

Damp targets of observed constraints in CM COLLISION PREVENTION

Figure 2: System overview

91

the total DOFs of the spine are reduced from 16 to 10 due to the coupling. Thus the computational cost is slightly less, and the results are more realistic as biomechanical considerations are taken into account.

Finally, damping may not be always desirable in the context of Virtual Reality interactions because it may conflict with the user’s actions. For this reason we propose the notion of user-controlled observer (§5.2) which is a user-chosen body part (typically the hand) for which collision avoidance is performed without damping. This lets the user fully responsible of its guidance whereas other body parts are automatically damped.

4

Handling large penetrations

The posture reconstruction layer of our system maps the motion of a human participant to the skeleton of a virtual mannequin. In our current implementation we use an active optical motion capture system (8 cameras) from Phasespace. The user is equipped with the reduced set of active markers (LED) shown on Figure 3.

Large penetrations may be present at the initialization stage that request a more computationally expensive resolution method. The Collision Resolution module takes care of such context. It does so by activating high-priority repulsive constraints to push away the body part from the obstacle, and iterating until the collision is solved or until a maximum number of iterations are reached. For both the posture correction and the collision resolution stages it is even possible that no collision-free posture can be found. Dealing with such a context would require a computational cost that our system cannot afford given the need for real-time frame rates [Baciu and Wong 2000].

2

Individual markers give us position information only. When grouped by two or three we are able to also infer the orientation of some body parts like the head, the wrists and the toes. This allows us to define the position and/or orientation goals of the kinematic constraints guiding the avatar posture over time. In addition, each constraint has been given a priority according to its relative importance in the recovery of a plausible avatar posture. For example the constraints on the toes have the most important priority whenever they are in contact with the floor. Otherwise, their priority is decreased on the fly during the on-line performance. Such an approach reduces foot sliding.

Multiple-Priority Inverse Kinematics

As shown in Figure 2, the posture reconstruction and collision avoidance layers of our system work by managing a set of constraints that determine the motion of the avatar. As these are sorted according to a hierarchy of priorities, the method we use to solve them must be able to account for these priorities. We have decided to use the multiple-Priority Inverse Kinematics technique (also referred to as PIK). This technique computes a posture variation of an articulated figure given the task vector ∆x that gathers increments towards the satisfaction of the constraints. The PIK architecture is based on the linearization of the set of equations expressing the Cartesian constraints x as functions of the joints’ degrees of freedom θ. By J we denote the Jacobian matrix gathering the partial derivatives ∂x/∂θ. Its pseudo-inverse, denoted J+, is used to build the projection operators PN (J) on the kernel of J, noted N(J). An efficient computation of projection operators allows splitting the constraints set into multiple constraint subsets associated with an individual strict priority level [Baerlocher and Boulic 2004]. The solution guarantees that a constraint with a high priority is achieved as much as possible. On the other hand, a constraint with low priority is solved only on the reduced solution space that is left after the satisfaction of all constraints of higher priority. Joint limits expressed as inequality constraints in joint space can also be enforced, at the highest possible priority level.

3

Posture Reconstruction with Prioritized Constraints

Figure 3: Active markers for recovering the user posture

Anatomically Coherent Avatar Postures

The controlled avatar is a standard hierarchical articulated structure rooted at the spine base with a Root node to allow free movement in space. We take into account the high coupling behavior of the human vertebrae by using a few sets of coupled joints strategically placed on the spine. In our system we use a model of three regions (lumbar, thoracic and cervical) in which the swing and twist components of rotation are coupled independently. This model is inspired by the work of Raunhardt and Boulic [2007]. The coupling relationships between the three regions can be expressed as equality and inequality constraints, and then integrated easily in the PIK solver. Equality constraints are enforced by modifying the initial Null Space projection operator PN (J), hence restricting the initial solutions space to those with coupled spine postures. On the other hand, inequality constraints are enforced in the same way as the joint limits, i.e. hard constraints are enforced whenever the computed solution violates a specified limit. By exploiting this spine model,

Marker n°

Constraint type

Controlled body part { recruited joint set }

Priority rank

18-19-20 / 21-22-23

position, orientation

both toes { until Root }

1

12

position

spine base { Root }

2

12-13-14

orientation

spine base { Root

3

0/4

position

wrists { elbow, shoulder and clavicle }

3

0-1 / 4-5

orientation

wrists { elbow twist }

4

2/6

position

both shoulders { clavicle }

5

3/7

position

both clavicles { until Root }

5

9 / 11

position

both knees { until Root }

6

8 / 10

position

both ankles { until Root }

7

15-16-17

orientation

head { cervical joints }

8

Table 1: Constraints used to reconstruct the participant’s posture

Besides, each constraint is allowed to recruit (i.e., use for its satisfaction) only a subset of the joints in the hierarchy. For example, the head marker constraints recruit only the cervical joints. The full set of constraints used by the posture reconstruction layer, together with their type (position/orientation), initial priority rank and recruitment level is summarized in Table 1. Each IK iteration aims to satisfy the kinematic constraints by producing a

92

yields, in case it is positive, a damping factor that serves to correct the motion of the observer.

posture variation of the avatar that is used to build the proposed pose in the collision prevention module (Figure 2).

Algorithm 1. PREVENTIVE CONSTRAINT ACTIVATION

5 5.1

Preventive Collision Avoidance Input: P, O, n, d, ∆o, ∆p, D Output: 1D preventive constraint or null if no damping is needed begin ∆reln ← (∆o − ∆p)⋅ n

General case

Observers The post-IK posture correction layer (Figure 2) is based on the key notion of observer, first introduced by Peinado et al [2005]. An observer is an entity that monitors the motion of the body part to which it is attached, reacting in order to prevent short-term collisions. In the following, for brevity, whenever we talk about an observer’s properties such as shape or position, we refer to the properties of its body part. The way an observer interacts with the environment depends on its shape (to simplify proximity queries, we approximate body parts with simple convex primitives such as line segments, spheres or capsules).

if ∆reln > 0 then # moving away from the obstacle return null else { λ←0 penetration ← false if d < D # Main test λ ← 1 − (d/D)2 endif

Computing the proposed pose

ddamped ← d − |∆reln|(1−λ) if ddamped < 0 then λ ← (|∆reln| − d) / |∆reln| penetration ← true endif

Each IK iteration starts on the current avatar pose and computes a new pose, which we call the proposed pose. The proposed pose may be partially damped by the pre-IK target correction layer, which applies a similar damping to the one described here, but at the constraint-target level, and only for those observers that have an associated position constraint (§1.2). The proposed pose is applied to a copy of the avatar’s skeleton. This differs from previous research in which body part displacements were approximated, for efficiency, using the Jacobian matrix returned by the IK solver [Peinado et al. 2005]. In the present work we have discarded this approach because the computation time saved is not as critical as the precision gained with a fully-updated separate skeleton.

# Secondary test

if λ = 0 then # no need to damp return null else # 1D constraint along n { g ← (1−λ)∆reln + ∆p⋅ n priority ← VARIABLEPRIORITY (d, D, penetration) type ← 1D position constraint along n return NEWCONSTRAINT(observer, priority, type, n , g) }

Relative displacements

} end

An observer acts by checking its position in the proposed pose, with respect to that in the current pose. If there is relative displacement towards a near obstacle, the observer creates a constraint to damp this displacement. When moving towards several obstacles at the same time, an observer reacts only to the closest one. Obstacles are allowed to move with any linear or angular velocity. For our algorithm, we exploit only the obstacle displacement over the current simulation step.

First, the main test simply checks whether the current position of the observer is inside the influence area of the obstacle. If so, we compute a damping factor given by λ = 1−(d/D)2. λ tends to 1 (full damping) as the observer approaches the obstacle, and to 0 (no damping) near the influence zone boundary.

Preventive constraint activation The specifics of preventive constraint activation are given in Algorithm 1, which is invoked for every observer once the IK solver has computed the proposed pose (Figure 4). The input parameters are the following: -

the observer current (resp. proposed) position Oc (resp. Op ), the point P on the obstacle closest to Oc, the normal vector n to the obstacle surface on P, the observer proposed displacement vector ∆o , the obstacle displacement vector ∆p, the signed distance d between P and Oc (positive when Oc is outside the obstacle), and the thickness D of the obstacle influence area. The algorithm begins by projecting the relative observer-obstacle displacement vector ∆rel onto the obstacle normal n, producing the scalar ∆reln. A positive amplitude means that the observer is moving away from the obstacle, so the observer is considered safe and the algorithm ends without activating any preventing constraint. Otherwise, we perform two successive tests. Each one

Figure 4: Damping in the influence area of a planar obstacle. For simplicity the observer is drawn as a point, although more complex shapes are supported.

Then, regardless of the result of the first test, we perform the secondary test to check whether the resulting damped displacement penetrates the obstacle. This test handles two critical

93

cases: 1) when the amplitude of the relative displacement is large relatively to the influence area thickness D, as in Figure 5, and 2) when the current position is itself penetrating the obstacle (Figure 6). For this we compute the damped distance ddamped, which is the signed distance of the resulting damped displacement along the normal n. If it is negative we compute a new damping factor λ producing a displacement exactly onto the surface of the obstacle.

5.2

User-Controlled Observers

In our system no observer set-up is assumed or preferred. The user is free to add observers to those body parts that are likely to collide given the kind of motions to be performed. Likewise, the specific shape assigned to each observer is decided by the user, who might opt for simplicity (e.g., a sphere to model the head) of for accuracy (a capsule). The only recommendation is that the number of observers is kept as small as possible in order to reduce the computation cost of observer-to-obstacle proximity queries. In addition to the regular observers that are automatically damped by the system, our system introduces the notion of user-controlled observer. The need for this entity stems from the observation that, during most common interactions, the attention of the participant is directed towards controlling a particular body part, which we call the focus body part. For a focus body part, the discrepancy introduced by the damping mechanism might be too distressing. In addition, any participant is assumed to be able to control such a “focus” body part in a collision-anticipative manner, making the introduction of further damping by our system undesirable. A user-controlled observer is special in that it does not apply any damping when approaching an obstacle. Therefore, the motion of the body part is not disrupted until it is absolutely necessary to avoid interpenetration. This behavior resembles the well-known rubber band method (Figure 1a). Note that this helps to enrich the level of interaction allowed by our system, as the participant is now allowed to “touch” the surface of obstacles using a usercontrolled observer. This would be difficult with a regular observer, which would be slowed down by damping as it got closer to the surface.

Figure 5: Due to a large displacement relative to D, the observer may penetrate the obstacle even after the main damping test (left). This is detected by performing a secondary test, which brings the observer exactly to the surface (right).

5.3

Self-collisions

Handling self-collisions is usually unnecessary in motion capture, as the avatar is under the control of the user full body. In our system, however, we correct the motion that is mapped to the avatar, and this correction may cause a self-collision. For example, arm motion is likely to get corrected during a near-obstacle reaching task, potentially resulting in moderate elbow-torso interpenetration. A straightforward extension of our system allows us to deal with self-collisions in a unified manner, as well as collisions between avatars. Each time an observer tests its associated body part for collisions with the environment, it treats other body parts as obstacles, and Algorithm 1 applies. A few additional issues have to be considered. First, those body parts which are found to collide in the reference pose are discarded from further tests (e.g., the fore and upper arms). Second, for efficiency the system keeps a predefined list of body part pairs that need not be checked against each other (e.g., the legs and the torso). Finally, we do not apply the relative displacement formulation to collisions between body parts. This way we prevent unnatural reactions, like the trunk shifting sideward to avoid a hand that moves nearby. In practice this simply means that we apply Algorithm 1 with ∆rel = −∆ο instead of ∆rel = ∆p–∆ο, and without adding the obstacle relative displacement term to g.

Figure 6: The observer current position may slightly penetrate the obstacle. The secondary test removes the penetration thanks to a repulsive (>1) damping factor (right).

Once the activation of the preventive constraint is detected through the non-null value of λ, we create a 1D constraint to restrict the motion of the observer along n (dotted arrows in Figure 4). Motion in the tangent plane is left uncontrolled. To obtain the 1D goal distance g we add the damping term to the obstacle relative displacement along the normal n. Handling small penetrations As discussed earlier small penetrations are possible for various reasons (linearization, conflicting constraints, limited number of online IK re-evaluations). The algorithm detects such case through the secondary test (Figure 6). The computed value for λ is greater than 1 which pushes the observer outwards onto the obstacle surface. Priority computation As multiple observers may simultaneously create preventive constraints, a priority is associated to each constraint based on their proximity to their respective obstacle. When the observer just enters an obstacle influence area, its priority can be in the same range as the marker constraints, but as the observer get closer to the obstacle it becomes more important than the marker constraints. The limit case of penetration correction constraint is associated with the first priority rank. In this way posture control is preferential when no obstacle is too close, and collision prevention is preferential otherwise.

6 6.1

Experimental Results Offline Tests

During the development phase we tested our system offline using synthetic motions generated with our IK solver. Figure 7a shows the output of one of these tests. The virtual character starts with a neutral posture, and has to reach a low target. Capsule-shaped observers are used it the different joints of the left arm. A large cylindrical obstacle is placed on the way to the goal, and avoided

94

appropriately thanks to the preventive constraints created by the observers. These constraints are displayed as lines connecting each observed body part to the obstacle.

Therefore we conclude that the RB model used on the focus body part only is not appropriate when full-body interactions are considered.

Our second experiment, intended to demonstrate the repulsion behavior that can be obtained with our relative velocity formulation, is simpler to set-up. The character is constrained to remain in a neutral pose, and a rotating cylinder is placed near him (Figure 7b). As the cylinder approaches, the virtual character dodges it while keeping as close as possible to the neutral posture. Note that the center of mass constraint is also enforced regardless of the damping mechanism. This is possible because we assign that constraint a priority which is above the range reserved to those activated by observers.

Test 2: full-body rubber-band (Figure 10). In our next test we extended the RB method so that other body parts are checked for collisions. The set-up is similar, the only difference being that we use capsule observers in body parts that are likely to collide, like the arms, the legs and the head. The result is improved in that no collisions take place. The problem is that the result is unnatural in that the avatar appears to lack a sense of planning. Instead, it seems to circumvent the obstacles by “trial and error”. We conclude that it is possible to extend the RB method by applying it to full-body interactions, but that a sense of anticipation can still be beneficial.

Running on a 1.8 GHz AMD Sempron processor with 1GB of RAM, the posture updates took an average of 7.1 and 5.8 ms, respectively. The first one is slower because more constraints must be satisfied by the IK solver.

Test 3: damping (Figure 11). We followed with a test to compare RB with our damping method. For that purpose we repeated the set-up of the previous test, but now RB was substituted by our damping formulation. For the left hand a user-controlled observer is used, for the reason explained in section 5.2. The output is collision-free as well, but now the damping exerted by the observers helps to keep clearance between the arm and the obstacles. This appears more natural than Test 2, in which contact occurred at several points. Since pre-IK target correction is used, this damping is achieved for many of the observers without the need to activate a preventive constraint (this implies improved performance).

(a)

Test 4: artificial safety distance (Figure 12). It could be argued that, without damping, a similar sense of anticipation could be achieved by artificially adding a “safety margin” around each obstacle. By not allowing observers to enter this margin, the resulting motion would present the sense of anticipation that was lacking in plain RB. To test this hypothesis we performed a new test similar to Test 2; observers are used in the limbs and in the head, and the RB method is used. The difference now is that the observers are not allowed to get closer than a margin Dsafety from obstacles. The results reveal that, in this scenario, the safety margin prevents the task from being performed, as the “extended” obstacles occupy all the space on the hand’s way to the goal. We therefore conclude that damping is a more flexible way to provide anticipative collision avoidance.

(b)

Figure 7: (a) Reaching with preventive collision avoidance. The lines represent activated observers. (b) Repulsion by a rotating obstacle

6.2

Online Tests

Test 5: two hands (Figure 13). In this final test the set-up is slightly different. There is only one obstacle, and the participant raises his hands two times. The first time he deliberately tries to penetrate the obstacle. This is prevented by the system but damping is not applied as the observers in the hands are tagged as user-controlled. In addition, the motion of the head is also corrected but this time with damping. The second time the hands are raised, the participant keeps a prudent safety distance.

After the offline validation, we tested our system in an online VE. The participant was equipped with the marker set described in section 4. This set is less encumbering than most production setups, yet complete enough for an accurate reconstruction of postures, including features like the swivel angle of the elbows. Motion was mapped to the avatar using a Prioritized IK solver and the set of marker constraints of Table 1. We used a retro projected display and a third person view which allowed the participant to capture the whole scene at one glimpse (the choice of the best viewpoint approach –first vs. third person– is beyond the scope of this paper). Our main goal in these tests was to validate our damping method by comparing it to alternative ways to deal with collisions. All of them can be better appreciated in the accompanying videos.

This demo illustrates the decoupling between the position and the orientation kinematic constraints. Even after an important visualproprioceptive discrepancy has been introduced, the avatar is still responsive to the orientation features of the motion performed by the participant. In particular, the twist motion of the wrists is accurately transferred to the avatar. The real pose provided by the markers is shown too for comparison with the corrected one, in a way that resembles the “ghost” metaphor used in previous work.

Test 1: baseline rubber-band method (Figure 9). In first place, we tested the traditional rubber band (RB) method. For that purpose, we implemented RB and tested it in the scenario shown in Figure 6a. The red cube is the goal to be reached. Capsule observers are attached to hands of the avatar. Completion of the task is hindered by two cylindrical virtual obstacles placed at different heights in front of the avatar. Since we are testing the RB method, no damping is applied. Instead, collisions are reacted to only after they take place. It can be seen that the RB produces collision-free motion of the hand. Besides, since the rest of body is not controlled by observers, collisions occur with the left arm.

We use Test 5 too as a means to evaluate the performance of our system. Three factors are measured: computation time per frame (rendering excluded), number of preventive constraints used, and accuracy of the participant-to-avatar motion transfer. The result is shown in Figure 8. It can be seen that the first time the hands are raised, a large number of preventive constraints are activated to avoid penetration. This is followed by an increase in the computation time (as the IK solver is forced to solve more constraints, and more iterations are necessary to achieve the

95

damping) and also by an increase in posture error, which is an indicator of the heavy correction applied to the hands. The second times the hands are risen less constraints are created (a maximum of three when the lower arms and the head are simultaneously damped), and the computation time and posture error are diminished accordingly.

the captured interactions of real participants. Our results indicate that, although adding collision avoidance imposes a significant performance penalty, the resulting system is nevertheless able to operate at real-time rates. More tests are needed to finely evaluate the computing cost as a function of the number of obstacles and of observers.

In any case, even on these frames when the highest number of constraints is used, the computation time per frame is low enough (90 ms) to permit a sufficient interaction frame rates. For the sake of comparison, Figure 8 reveals that the computation time of the posture reconstruction layer alone (i.e., when no preventive constraints are active) is about 20 ms per frame. The cost of an IK iteration is quadratic with the constraints’ dimension M. Given that the preventive constraints’ dimension is either null or remains a small fraction of M, the cost of the full algorithm mostly depends on the number of IK evaluations in a linear fashion (limited to two re-evaluations in the present implementation). In comparison, the time spent doing proximity queries is negligible.

In terms of sense of immersion and task performance, our current evaluation of the system is based only on the comments provided by the users during demonstrations. We plan to conduct formal user studies to derive more definitive conclusions. For example, we would like to know to what extent is the user tolerant to motion corrections, especially when motion is induced on his avatar to dodge an approaching obstacle as in Figure 7b. Another issue we plan to address is how the choice of perspective affects the user experience in the context of our collision avoidance system. We believe that the visual-proprioceptive discrepancy introduced by our system would be more disruptive using a first person perspective device (such as an HMD), but this point must be backed with experimental data. In third place, experiments could be conducted to support our assumption that user-controlled observers improve the level of interaction with respect to a system that uses damping alone. Currently a time-consuming part of our system is deciding which observers to use for some given environmental and task conditions. We plan to develop an algorithm for estimating a good set of observer given a particular task and a particular environment. It must be noted that the damping mechanism produces good results for simple interactions, like reaching or manipulating. Slightly more complex behaviors like squatting or stepping forwards can also be handled, but the system is not applicable to the broad variety of potential human behaviors. For instance, damping cannot correct realistically a motion of the avatar walking to enter an obstacle. For such scenario, other approaches (such as locomotion planning) would be necessary. On a related note, our relative velocity formulation allows only for basic obstacle dodging movements. More complex psychologically-inspired maneuvers would require high-level decision making [Metoyer et al. 2007.] For example, an obstacle approaching the shins of the avatar cannot be dodged realistically unless a “jump” or “step aside” controller is activated. This is outside the scope of our method.

Figure 8: Performance indicators for Test 5. The test was conducted on a 3.2 GHz Intel Xeon processor with 1 GB RAM. Computation times (in ms) do not include rendering. Posture error is defined as the RMS error for all marker constraints (in meter). The green step function indicates the current number of activated preventive constraints.

7

Conclusions and future work

Another limitation in our system is that the damping method deals with obstacles separately, whereas an algorithm that takes into account the environment as a whole could help solve a wider range of scenarios. We are currently working to extend the algorithm for handling an arbitrary number of overlapping obstacles, including concave obstacles. Also, our system would benefit from the support for higher-order constraints (dynamics), although it is not clear how our prioritized formulation could be extended in that direction while operating at real-time frame rates

We have presented a system that exploits the Prioritized Inverse Kinematics technique to achieve a real-time mapping of full-body postures from a human participant to his avatar in a virtual environment. When no obstacles are near, this mapping is accurate. Otherwise, a collision avoidance layer which exploits the IK solver uses preventive constraints to correct the participant’s motions as they are transferred to the avatar. This correction is applied in two related ways. First, the displacement of observed body parts towards nearby obstacles is damped to prevent potential collisions. Second, an approaching obstacle induces repulsion in near body parts, leading to human-like “dodging” reactions. Both effects are achieved with a single formulation in terms of relative observer-to-obstacle velocities. Large observer or obstacle displacements are handled thanks to a secondary test, which we use to deal with contact situations as well. To account for collisions in the initial posture or for collisions that arise when a new obstacle is added to the VE, our system incorporates a repulsion mechanism which uses highpriority constraints to push colliding body parts away from obstacles.

Acknowledgements This research has been partially supported by the EU ENACTIVE Network of Excellence, the Swiss National Science Foundation under grant 200020-109989 and by the University of Alcalá under grant PI2005/083. We would also like to thank Mireille Clavien for designing the virtual human model and the anonymous reviewers for their valuable suggestions.

References BACIU, G., WONG, S. K. 2000. The Impulse Graph: A New Dynamic Structure for Global Collisions. Computer Graphics Forum 19(3), 229-238.

We have provided offline experimental validation of our system, using synthetic motions. We have also tested it online, to correct

96

BADLER, N.I., HOLLICK, M.J., GRANIERI, J.P. 1993. Real-Time Control of a Virtual Human Using Minimal Sensors. Presence: Teleoperators and Virtual Environments, 2, 82-86.

LEE, J., CHAI, J., REITSMA, P.S.A., HODGINS, J.K., POLLARD, N.S. 2002. Interactive control of avatars animated with human motion data. In Proc. Of SIGGRAPH '02, 491-500.

BAERLOCHER, P., BOULIC, R. 2004. An Inverse Kinematic Architecture Enforcing an Arbitrary Number of Strict Priority Levels. The Visual Computer, 20(6), 402-417.

LINDEMAN, R., SIBERT, J., TEMPLEMAN, J. 2001. The Effect of 3D Widget Representation and Simulated Surface Constraints on Interaction in Virtual Environments. In Proc. of the 2001 IEEE Virtual Reality Conference, 141-148.

BERG, J., FERGUSON, D., KUFFNER, J. 2006. Anytime path planning and replanning in dynamic environments. In Proc. of the IEEE International Conference on Robotics and Automation (ICRA ‘06), 2366-2371.

VAN DEN

LOK, B. 2002. Interacting with Dynamic Real Objects in Virtual Environments. PhD thesis, University of North Carolina at Chapel Hill.

BORST, C.W., INDUGULA, A. P. 2005. Realistic Virtual Grasping. In Proc. of the 2005 IEEE Virtual Reality Conference, 91-98.

MACIEJEWSKI, A.A., KLEIN, C.A. 1985. Obstacle Avoidance for Kinematically Redundant Manipulators in Dynamically Varying Environments. International Journal of Robotics Research, 4(3), 109-117.

BOWMAN, D., HODGES, L.F. 1997. An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments. In Proc. of the 1997 Symposium on Interactive 3D Graphics, 35-38.

METOYER, R.A., ZORDAN, V.B., HERMENS, B., WU, C.C., SORIANO, M. 2007. Psychologically inspired anticipation and dynamic response for impacts to the head and upper body. Transactions on Visualization and ComputerGraphics (to appear).

BROCK, O., KHATIB., O. 2002. Elastic Strips: A Framework for Motion Generation in Human Environments. International Journal of Robotics Research, 21(12), 1031-1052.

PEINADO, M., BOULIC, R., LE CALLENNEC, B., MEZIAT, D. 2005. Progressive Cartesian Inequality Constraints for the Inverse Kinematics Control of Articulated Chains. In Proc. of Eurographics 2005, Short Presentations Session, 93-96.

BURNS, E., RAZZAQUE, S., WHITTON, M.C., MCCALLUS, M.R., PANTER, A.T., BROOKS, F.P. 2006. The Hand is More Easily Fooled than the Eye: Users Are More Sensitive to Visual Interpenetration than to Visual-proprioceptive Discrepancy. Presence: Teleoperators and Virtual Environments, 15, 1-15.

PEINADO, M., BOULIC, R., RAUNHARDT, D., MEZIAT, D. 2006. Environmental-aware Postural Control of Virtual Humans for Real-time Applications. In Proc. of the 2006 SAE Digital Human Modeling Conference.

BURNS, E. BROOKS, F. P. 2006. Perceptual sensitivity to visual/kinesthetic discrepancy in hand speed, and why we might care. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (Limassol, Cyprus, November 01 - 03, 2006). VRST '06. ACM Press, New York, NY, 3-8.

RAUNHARDT, D., BOULIC, R. 2007. Exploiting Coupled Joints. In Proc. of 2nd International Conference on Computer Graphics Theory and Applications (GRAPP ‘07). ROSENBAUM, D. A., MEULENBROEK, R. J., VAUGHAN, J. 2001. Planning reaching and grasping movements: theoretical premises and practical implications. Motor Control, 5(2), 99115.

FIORINI, P., SHILLER, Z. 1993. Motion Planning in Dynamic Environments Using the Relative Velocity Paradigm. In Proc. of the 1993 IEEE Int. Conf. on Robotics and Automation, Vol. 1. 560-565.

SHAPIRO, A., KALLMANN, M., FALOUTSOS, P. 2007. Interactive Motion Correction and Object Manipulation. In Proc. of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D).

GARBER, M., LIN, M.C. 2002. Constraint-based Motion Planning Using Voronoi Diagrams. In Proc. of Workshop on Algorithmic Foundations of Robotics.

STURMAN, D.J. 1998. Computer Puppetry. IEEE Computer Graphics and Applications 18(1), 38-45.

GOVINDARAJU, N. K., LIN, M.C., MANOCHA, D. 2007. Efficient Collision Culling among Deformable Objects using Graphics Processors. Presence: Teleoperators and Virtual Environments,15(1), 62-76.

TOLANI, D., GOSWAMI A., BADLER, N.I. 2000. Real-Time Inverse Kinematics Techniques for Anthropomorphic Limbs, Graphical Models 62(5), 353-388.

KIM, Y. J., REDON, S., LIN, M. C., MANOCHA, D., TEMPLEMAN, J. 2007. Interactive Continuous Collision Detection Using Swept Volume for Avatars. Presence: Teleoperators & Virtual Environments 16(2), 206-223.

ULLMANN, T., SAUER, J. 2000. Intuitive virtual grasping for nonhaptic environments. In Proc. of the 8th Pacific Conference on Computer Graphics and Applications, 373-379. ZHAO, X., BADLER, N. 1994. Interactive Body Awareness. Computer-Aided Design, 26(12), 861-866.

Figure 9: Test 1. The rubber band method is applied to the hands only, leading to collisions in the arm. The magenta area around obstacles has no meaning.

97

Figure 10: Test 2. The rubber band method is applied to the full body. The magenta area around obstacles has no meaning in this test.

Figure 11: Test 3. Our damping system is used for collision avoidance. As a consequence, a safety distance is preserved between the arm and the obstacle. The influence area of the obstacles is shown in magenta.

Figure 12: Test 4. The full-body rubber band of Test 3 is extended with a safety distance around obstacles (shown in magenta). The task cannot be completed.

Figure 13: Test 5. Both hands are moved in front of a cylindrical obstacle (shown as wireframe for clarity). The participant starts on a reference posture (frame 1). He then raises his hands until they penetrate the obstacle, and this penetration is avoided by our system. The real participant posture is shown as a “ghost” (frame 2). When the danger of collision disappears, the preventive constraints are removed and the discrepancy between the real and the corrected postures goes away (frame 3). The hands are then raised again, although the correction is less important (frame 4).

98