Proceedings of 2004 IEEEIRSI International Conference on Intelligent Robots and Systems September28 - October 2,2004, Sendal,J a p n

Robustness of Central Catadioptric Image-Based Visual Sewoing to Uncertainties on 3D Parameters Youcef Mezouar LASMEA, Universit6 B. Pascal 63177 AubiBre Cedex, France [email protected]

Abstract-Tbb paper concerns the stability analysis of image-based visual servoing metbods with respect to uncertainties on the 3D parameters introduced in the mntral catadioptric interaction matrix. Motivated by the growing inter& for omnidirectional sensors on robotic applications and particularly on vision-based control, we extend reeent results obtained for conventional cameras to the entire elass of central catadioptric systems. In this paper, it is shown that with such sensors extreme care must be taken when approximating3D parameters to emure stabilityof L e imagebased mntml law.

I.

INTRODUCTION

In the last years, the use of visual observations to control the motions of robots has been extensively studied (approach referred in the literature as visual servoing). This is known to be a very efficient and flexible method for applications such as the positioning of a robot and the tracking of objects using the visual informations provided by an in-hand camera. Three groups of vision-based control laws have been proposed in the literature namely position-based, imagebased and hybrid-basedcontrol[91, [ 1 I], [12]. Contrarily to model-based visual servoing methods, image based visual servoing does not need the knowledge of the full model of the target. On the other hand, it is necessary to provide some information about depths of the object in the camera frame. Until recently, it was generally believed that a rough approximation of the depth distribution was sufficient to ensure the stability of the control law and most of the effort has been concentrated on the solution of convergence problems. Indeed image-based visual servoing is a local control solution which, even under perfect modeling can fail if the initial camera displacement is too big. Some methods have been proposed to address this issue based on the choice of features sharing good properties [IS] or based on path planning 1151. Nevertheless, if the environment is completely unknown and the robot is uncalibrated the stability of the visual servoing in presence of depth estimation errors, can become a serious issue. In [13], the stability of image-based visual servoing with respect to depth distributions ~ I I M S has been studied. It has been shown that the robusmess domain is not so wide when conventional camera is used. In this paper, we extend the stability analysis to the entire class of central catadioptric

0-7803-8463UJO4n20.00 WO04 IEEE

Ezio Malis INRIA, Sophia Antipolis 2004, route des Lucioles, France

[email protected]

cameras which includes the conventional ones. This work is motivated by the growing interest for omnidirectional sensors in robotics application. Indeed, conventional cameras suffer from restricted field of view. Many applications in vision-based robotics, such as mobile robot localisation [4] and navigation [ZO],can benefit from panoramic field of view provided by omnidirectional cameras. Clearly, visual servoing applications can also benefit from cameras with a wide field of view since such methods make assumptions on the link between the initial, current and desired images. They require correspondencesbetween the visual features extracted from the initial image with those obtained from the desired one. These features are then tracked during the camera (andor the object) motion. If these steps fail the visually based robotic task can not be achieved [6]. Some methods have been proposed to resolve this deficiency based on path planning [141, switching control [7], zmm adjustment [171, geometrical and topological considerations [81. However, such strategies are sometimes delicate to adapt to generic setup. Omnidirectionalcameras naturally overcome this problem. As a consequence, such sensors has been successfully integrated as part of a closed loop feedback control system [51, [161, [191, [ZI. In the literature, there have been several methods proposed for increasing the field of view of cameras s y ~ tems [3]. One effective way is to combine mirrors with conventional imaging system. The obtained sensors are referred as catadioptric imaging systems. The resulting imaging systems have been termed central catadiopmc when a single projection center describes the world-image mapping. From a practical view point, a single center of projection is a desirable property for an imaging system [I]. Baker and Nayar in [l] derive the entire class of catadioptric systems with a single viewpoint. In this paper, we study the robustness of image-based methods with respect to errors on the 3D parameters introduced in the interaction matrix when a central catadiopmc camera is used as sensor. 11. THEORETICAL BACKGROUND

In this section, we describe the projection model for central catadioptric cameras and then we focus on eyein-band image-based visual servoing methods.

1389

generic projection model [lo]:

Taking into account the intrinsic camera and mirror parameters, the 3D point X is projected in the real image plane to a point with homogeneous pixel coordinates p = (U, v, 1): p =KMm

(2)

where the upper triangular matrix K contains the camera intrinsic parameters, and the diagonal matrix M the mirror intrinsic parameters: Fig. 1.

Generic camera model

0

1P-E

M=[ A. General camera model

0 0

q-t 0

1

Notice that a catadioptric sensor with a planar mirror is equivalent to a-conventional perspective camera. Using approximations K and M of the camera and mirror intrinsic parameters K and M and a measured image point p it is possible to compute the corresponding normalized point from equation (2): 6 = M-’K-’p. Obvionsly if the %amera and mi1121 intrinsic parameters are perfectly known K = K and M = M the normalized coordinates are perfectly estimated 6 = m.

As already mentioned in the introduction, a single center of projection is a desirable property for an imaging system. A single center implies that a point in 3D space projects into a single point in the image plane. Conventional perspective cameras are single view point sensors. As shown in 111, a central catadioptric system can be built by combining an hyperbolic, elliptical or planar mirror with a perspective camera and a parabolic mirror with an orthographic camera. To simplify notations conventional perspective cameras will be embedded in the set of central catadioptric cameras. In [IO], a unifying theory for central panoramic systems is presented. According to this generic model, all central panoramic cameras can be modeled by a central projection onto a sphere followed by a central projection onto the image plane (see Fig. 1). This generic model can be parametrized by the couple (E, p) (see Tab.1 and refer to [ZI)describing the type of sensor and the shape of the mirror. Setting 5 = 0, the general projection model becomes the well known perspective projection model.

B. Inreracrion matrix of cenrral caradioprric cameras Consider a 3D point Xi with coordinates Xi = ( X i , Y ; , Z , )with respect to 3, and its normalized image coordinates extracted from m: si = ( x i ,y i ) . The derivative of si with respect to time is:

si = LIV where v is the velocity of the camera and Li is the interaction mauix. The interaction matrix can be derived by differentiatingthe function f(X with respect to the camera pose evaluated at the origin or y followin~[2].Li can he decomposed into two sub-matrices L, = [ , Bi]with :

b

+

TABLE I CENTRAI. CATADIOPTRIC CAMERAS DESCRIPTION: 5.ah. bh. a., b. depend only of the r “ r inmimic p m c t c r s d

and p

Let 3cand 3, be the frames attached to the conventional camera and to the mirror respectively. Suppose that 3cand 3, are related by a translation along the Z-axis. The centers C and M of Fc and 3, will be called optical center and principal projection center respectively. Let X be a 3D point with coordinates X = [ X Y ZIT with respect to Fm. After setting p = J X 2 Y 2 Z2, let m = (5,y , 1) be the point (in normalized homogeneous coordinated) projected into a virtual plane according to the

+ +

where ui = J1+ (1 - E2)(x: y ; ) . Equations (3) and (4) present the general central catadioptric interaction matrix as a function of image coordinates si, the distance pi = J X : y2 Zf and sensor parameter E. Notice that if E = 0, then the interaction matrix Li is the well known interaction matrix for conventional perspective cameras.

+ +

C. Imge-based visual servoing The goal of imaged-based visual servoing is to position a robot by controlling the current position of the robot such that the current measured image features s reach their reference s*. Consider the following task function 191:

-

e = L+(s - s’)

1390

E+

uauon (9) shows that q6 can he computed as a function %image coordmates si and sensor parameter E. The mamx Ai can thus he rewritten as:

where is the pseudo-inverse of an approximation-of the true (211x6) interaction matrix. In [91, the matrix L+ is supposed to he constant while in this paper we consider the most general case when the matrix is not constant. In that case, the derivative of the task function is:

&+

e = -(S

dt

- 8')

-

+ L+S = (o(S-

S')

+ E'L)

whereO(s-s')isa6~6matrixsuchthatO(s-s*)l.=..

V

(5)

Note that, only the depth Z, is unknown in the matrix Ai.Ifweconsiderthe (2nxl)vectors = ( S I , S ~ ; . . , S ~ ) . the corresponding (Zn x 6) interaction matrix is L(z, s) = (L1, Lz,. .. , Ln) and the time derivative of s is:

=

0. Consider the following control law: (6)

v=-Xe

s = L(2, s)v

In order to compute the control law it is ycessruy to provide the approximated interaction matrix L. Plugging

Matrix L(z, s) can he decomposed into two (2% x 3) suh-

equation (6) into equation (5), we obtain the following closed-loop equation:

-

e = - X ( 0 ( 8 - s') -I- L+L)e

matrices:

L(z,s) = [A(z,s) B(s)l

(7)

e = XQe

A(z, S ) = D(z) C(S) where:

where Q = -LtLI,,,.. The linear system (8) is aymptotically stable if and only if Q has eigenvalues with negative real part:

is a (2nx2n) diagonal matrix containing the depth disaihution z.

0, where 4 1 1 is stable if and only if F is stable. Therefore, we can focus on the stability of F.

C. Necessary and sufficient conditions The eigenvalues of F are the roots of the characteristic polynomial:

X3 - tr(F)Xz

Z,

= d/nTrn

n

1 + -(tr(F)' 2

- tr(Fz))X - det(F) = 0

where U and det are respectively the trace and the determinant of a manix. The necessary and sufficient conditions for the roots of the polynomial to have negative real part are obtained from the Routh-Hunuitz Theorem:

u(F) < 0 < 0

e(FZ)-tr(F)* det(F) tr(F)(tr(F)' - u(Fz))- 2det(F)

< 0

< 0

The necessary and sufficient conditions can be used to test the stability of the servoing and to obtain the robustness domain (see for example the simulations in section IV). However, for a large number of parameters

where n is a unit vector which is a function of two parmeters n(O,$) = +(@)sin(4),sin(B)sin(b),cos($)). The estimgeg d e p t h i can be obtained using an approximation of q e , 4) and d: Z, = d/iiTm

-

$e

= n T m 3, then: 7, = @ = " m. and $i= Zr 7j As expected,' the-stabihty of the visual servoing does not depends on d but only on 6. Figure 2 to.Figure 5 show the stability regions for conve?tio$, parabolic and hyperbolic cameras as a function of (8,++)for an increasing number of points on the same plane. The true normal is n = (0.5,0,0.866) (i.e. B = 0 and 6 = ~ / 6 )In . the green region all the eigenvalues are negatives, the system is locally asymptotically stable. In the red region at least one eigenvalue is positive, and the system is locally unstable. Finally, the normals obtained in the blue region are discarded since we obtain at least a negative depth, which are cases not considered bere. First, note that all central cameras have similar stability region. Thus one can not expect to increase significantly the stable region by changing the type of central camera. When considering 3 image points (see Figure 2), the corresponding stable region is not so wide. Note that, adding a point inside the triangle defined by the others points (see Figures 3) only slightly modifies the stability region. Thus, when learning the reference image, one can think that it is probably better to chose points spread in the image. Unfortunately, the stability analysis shows that it is not always true. Indeed, if we add 4 more points as in Figure 4, the stability region in green is even reduced. Note that, if we have absolutely no idea on the 3D position of the plane, a simple guess ii = (0,0, 1)makes the visual servoing unstable. On the other hand, if many points are well distributed in all the image as in Figure 5 the stability

1392

REFERENCES

--

--

(a) Translation velocity

(b) Rotation velocity

Fig. 6. Unstable image~basedvisual servoink with conventional e a "

-(a) Translation velocity

@) Rotation velocity

Fig. 7. Unstable image-based visual servoing with parabolic -era

[I] S. Baker and S. K. Nayar, A theory of single-viewpoint cWopwic image formation. Int. Joumol o j Compurcr Rsion, 35(2):1-22, November 1999. I21 1. P. b t a . f? Martin, and R Horaud. vlsual servaing/tracking using eenoal catadioptric images. In 8ih Int. Symp. on Eiprimcntol Robotics, ISER'OZ, pages 863-869, Bombay. India, July 2002. [3] R. Benosman and S. Kang. Panommic Msion. Spinger Verlag. 2000. [4] P. Blaer and P.K. Allen. Topological mobile robot lwalilation using fast virion Vchniques. In IEEE h r . Conf on Robotics ond Automoiion, ICRA'OZ, pages 1031-1036. Washington, USA, May 2W2. (51 D. Burr% J. Geiman. and G.Hager. Optimal landmark eonfigud o n for vision based contra1 of mobile robot. In 18EE Int. Conf on Robotics ond A~tomodon,ICRA'03, pages 3917-3922, Tapei. Taiwan, September 2W3. [6] F. Chaumette Potential problems of stability and convergence in image-based and pasition-based visual servoing. Z k Con@" of Rsion and Conrml, W C l S Se&s, Springer Vedng, 237:6678, 1998. 171 0. Chesi, K. Hashimoto, D. Prauichino. and A. vlcino. A switching mnvol law for &ping fearures in the B eld of view in eye-in-hand visual smoing. In IEEE At. Comf on Robotics ond Automation ICR4'03, pages 3929-3934, Taipei, Taiwan, September 2003. [XI N. 1. Cowan, 1. D. Weinganm and D. E. Kodi(schek. Wsual ~ervoingvia navigation functions. IEEE Tmnr on Robotics and Amomtion, 18(4):521-533, August 2002. 191 B. Espiau, F. Chawnctte, and P. Rives. A new a p p c h Lo visual ~emoinein robotics. IE'EE Tmm. on Robotics and Automation, 8(3):313-326, June 1992. [IO] C. Geyer and K. Daniilidis. A unifying thwry fm central panoramic systems and practical implicaions. In Euumpeon Conf on Conpuler Rsion, ECCV'OO, volume 29, pages 159-179, Dublin, Ireland, May

region considerably increase. However, even in this very 2Mo. favorable case, there exist an important red instability [ll] S. Hutchinson. G.D.Hager, and P.I. Cork. A tutorial on visual servo region. control. IEEE Tmns. on Robotics and Automotion. 12(5):651470. In the second set of experiments (refer to Figures October 1996. [ 121 E. Malis. F. Chaumcrtc, and S. Boudct. 2 In d visual S C N O h g . IEEE 6 and 7). an image-based positioning task of Tmm. on Robotics ond Aulomnrion, 15(2):23&250, April 1999. a six degrees of freedom manipulator has been 1131 FA M 9 s and P. Rives. Robustness of image-based visual serming simulated. The initial camera displacement is very with respect to dcplh distribution mn. In IEEE In:. Confir" on Robotics and A u t o m i o n ICR4'03. T i m i . Taiwan Satember = -[0,001 0,001 0 , O O l l meters and small t 2W3. 7' = -[0,48 0,96 1,441 degrees. The m e depths are I141 Y,Mezouar and F. Chaumelle. Path planning for robust image-based z* = (0.9443 0.8470 0.9417 0.8498 0.9540 0 . 9 4 1 2 ) ~ ~ conv01. IEEE Tmns. on Roboncs ond Auromnrion, 18(4):534-549, August 2002. while the estimated depths distribution is 2 = [15] Y. Mezouar and F. Chaumelle. Optimal camera oalectory with (0.9500 0.8500 0.9500 0.8600 0.9460 0.9300) m. Despite itmgc~hasedconlml. Int. Joumol ojRoboricr Research,22(10):781the maximal error on the estimated depths is only 1,2% 804, October 2W3. of the true depth, one eigenvalue is, positive for aU central 1161 A. P a m o and H. h u j o . Multiple robots in geometric formation: Conml smucturc and sensing. In Int. Symp. on Infelligenf Robotic cameras (due of lack of space only figures for conventional System, pages 103-112, Reading, UK, July 2000. and parabolic cameras are present). Thus, even starting [I71 E. Malis S. Benhimane. Wsion-based conml with respect to planar and non-planar objccls using a zooming camera In IEEE Inl. Conf very close to the reference position, after iteration 350 Advanced Robotics. ICAR'W, pages 863-869, July 2W3. for the conventional camera (resp. 300 for the parabolic [I81 on to 0. Tahri and F. chaumcnc. Application of moment in-ants camera) the translation and the rotation errors start to visual servoing. In IEEE Inl. Conf on Roboiics and Automotion, ICRA'M, pagcs 42764281. Taipeh, Taiwan, May 2003. grow. In the beginning, the control law seems to be stable 1191 R. Wdal, 0. Shakcmia and S. Sasny. F o m t i a n o n l m l of since the other dominant eigenvalues have negative real nonholanomic mobile mboo with omnidirectional visual servoing and motion segmentation. In IEEE I N . Conf on Roboricr and Part. ~~~~~

V. CONCLUSIONS

In this paper, we have shown that extreme care must

Automorion, lCRA'03, Taipei, Taiwan, September 2W3. 1201 N. Winter, 1. Gaspar, G. Lacey. and 1. Santos-vlctar. Omni~ diRetional vision far robot navigation. In IEEE Workrhop on Omnidinctiowl Virion. OMNNIS'W, pgcs 21-28, South Camlina USA, June ZW0.

be taken when approximating 3D parameters of a target

for image-based visual servoing with central catadioptric cameras. Indeed, for these sensors the stability region in presence of errors on 3D parameters is not very large. It has been noticed that for all these sensors the stability region is similar and thus one can not expect to enlarge it by simply changing the type of central camera used.

1393

Fig. 2.

(a) lmage points with conventional camcm: Smhility regions: (b) conventional, (c) parabalic.

(b)

(a) Rg. 3.

(4

(d

(b)

(a)

(d) hyperbolic cameras

(4

(C)

(a) Image points with conventional camera: Stability regions: (h) conventional. (c) parabolic. (d) hyperbolic cameras

n

7 ; . ' '

...

- .

3

. . :

-. .

..

.

!

.. . . . . . . . ~

3 -I

'

'

..

'

-

.

'

''

.. .

(a)

- - --_

.

. .

..

(b)

(C)

.

l

e

(d)

Fig. 5. (a) lmagc points wirh conventional camera: Smhiliry regions: (h) conventional. (c) parabolic, (d) hyperbolic cameras

1394