OBSTACLES AVOIDANCE BY MONOCULAR MULTI-CUE IMAGE ANALYSIS S. Wybo, D. Tsishkou, C. Vestri, F. Abad, R. Bendahan, and S. Bougnoux Perception and Intelligence Department, Vision Group IMRA Europe S.A.S. 220 Rue Albert Caquot, B.P. 213, 06904 Sophia-Antipolis Cedex. FRANCE +33-493-957-388, [email protected]

ABSTRACT As was proved in the recent years and especially by the recent success of the DARPA Urban Challenge, autonomous navigation for vehicles is now possible even in difficult conditions. Unfortunately, those vehicles operating autonomously currently need combinations of multiple sensors such as laser scanners, radars and cameras to perform navigation. In this paper, we propose a combination of monocular vision based algorithms that enable obstacle avoidance at short range. Our approach combines structure from motion, movement detection and object recognition algorithms to efficiently detect obstacles. Using this system, we successfully demonstrated collision warning onboard a prototype vehicle and autonomous navigation (during a 4 hours run) of a robot in an unknown environment.

KEYWORDS Empty-space detection, Structure from motion, Movement detection, Objects recognition, Obstacle avoidance, Autonomous vehicle

INTRODUCTION Autonomous driving is the Holy Grail of all ITS engineers. Many projects have tackled this issue with increasing success, from Prometheus project (1) (2) and NavLab vehicles (3) to the recent performances of the DARPA challenges competitors (4) (5). As can be seen in Figure 1, these projects have relied on use of many high performance sensors (laser scanners, radars, stereo vision…) to reach their high performance. Unfortunately such setups are still a long way from being usable in a customer vehicle mostly because of their price. Rather than competing with these setups we have decided to shoot for short range collision avoidance using a widely available sensor: CCD camera. We aim at demonstrating that even such a standard setup (many recent car models have been fitted with a back wide angle camera) could help prevent many collisions with diverse obstacles.

-1-

Figure 1: DARPA Grand and Urban challenges winners use many very expensive sensors

In this paper we propose a solution to perform low speed obstacles avoidance and anticollision using only monocular vision algorithms. Obviously we don’t claim the same kind of performance that was achieved at the different DARPA challenges. Yet, we successfully demonstrated use of these algorithms in an in-vehicle anti-collision system as well as to perform autonomous navigation in an unknown indoor environment. In both situations many different obstacles were present: vehicles, humans and other random objects. We will first describe the development vehicles used, and then focus on the different algorithms developed: - Empty-space detection. - Structure from motion for static obstacles detection. - Moving obstacles detection using ground plane transformation. - Vehicles and humans recognition. Then we will briefly present the obstacle avoidance strategy we used. Finally, we’ll present the experimental results obtained.

DEVELOPMENT VEHICLES For a couple of years now we have been using a prototype vehicle equipped with a monocular vision sensor and odometry sensors offering 3D vision reconstruction of the surrounding static environment. This vehicle is a Toyota Picnic which we already presented in (6). It is equipped with a standard NTSC camera, odometry and steering-wheel angle sensors (Figure 2). It has no actuators necessary for autonomous driving.

-2-

Figure 2: Prototype vehicle setup

According to the growing demand for autonomous control we have recently started using a robotic platform in addition to our prototype vehicle. The robot we use is based on a heavily modified version of MobileRobots, Inc. PeopleBotTM platform. The platform was modified so that it could power the computer embedded and equipped with 2 standard CMOS cameras at its top front, one looking down for small and close objects, the second one looking for high and tall objects as shown in Figure 3. Additionally, the ultrasonic sensors that equip the standard platform were disabled and covered by the robot shell. As a result the robot can only use the information gathered from its cameras and from its wheels odometry similarly to our prototype vehicle. Image processing algorithms (and autonomous control on the robot platform) are performed by a high speed computer equipped with a quad core 3GHz CPU and 2GB of memory. Image acquisition on the robot is done through standard IEEE 1394a ports while it is done using a Matrox Meteor II frame grabber in the vehicle. In both cases full NTSC frames are captured at 30 frames per second. Odometry is grabbed through National Instruments acquisition board in the vehicle. In the robot, wheels sensors data is acquired through RS232 port as imposed by the platform design and used by a homemade odometry algorithm to provide precise positioning. This link is also used to control the robot movements by sending basic orders (angular and longitudinal speed orders) to the embedded robot micro-controller.

-3-

Figure 3: Design of the autonomous robot

MONOCULAR VISION-BASED ALGORITHMS In order to fit as closely as possible to current customer vehicles’ standard equipment we have chosen to use only monocular algorithms as single cameras are already available in many vehicles sold today while stereo-vision rigs are far less common. In this paper, we will focus on obstacles detection as this is the first mandatory stage for autonomous navigation. We combined multiple monocular vision algorithms to make detection more robust to the different kind of obstacles that can be encountered on the road. Three categories of algorithms were combined to make final decision: - Emergency control algorithm to check that there is no obstacle on vehicle path (free space detection algorithm). - Generic algorithms to detect, position and characterize (in terms of shape) static and moving obstacles (structure from motion and movement detection algorithms). - Dedicated algorithms to recognize specific objects which are most commonly found (human and vehicle recognition algorithms). Free space detection Although we aim at detecting all obstacles soon enough to avoid them it is necessary to provide autonomous vehicles with a generic mean to immediately stop in order to avoid a very close obstacle (which hasn’t been detected before or which just entered the scene very close to the vehicle). Free space detection just provides this mean by detecting any obstacle -4-

very close to the vehicle without trying to characterize it (as the 3D shape of an obstacle is not important for an emergency stop). The free space detection algorithm is a variation of the flood fill algorithm classically used for road detection. In this version of the algorithm the color homogeneity criterion generally used with flood fill is replaced by a texture similarity criterion. A 3 bin HSV histogram is used to capture texture properties in 8x8 pixels blocks to produce results robust to camera noise. By detecting vertical edges (computed through probabilistic Hough transform(7)), large vertical objects are searched and used as a cue to limit free space texture propagation. Finally a 2D/3D discrimination algorithm is used to discriminate real obstacles from flat ground marks. This later algorithm relies on the analysis of shadows for each potential 3D objects. Objects with a shadow are considered as 3D obstacles (thus stopping free space propagation) whereas objects without shadows are considered as marks on the ground which can be run over (free space propagation is not stopped on such objects). Figure 4 illustrates how intensity variations are used to assess the existence of a 2D or 3D object. If object is only 2D its intensity profile is different from that of a 3D object which shadow locally reduces this profile.

Figure 4: Illustration of 2D/3D discrimination technique for a 2D (red) and 3D (blue) objects

Eventually a map representing the free space in the image is generated. Analysis of the amount of free space in front of the vehicle combined with camera calibration provides information on the distance to obstacles. Too short distances to the obstacles in the vehicle path result in an emergency stop to avoid collisions. This algorithm has proved to be very efficient in areas with uniform/homogeneous road texture and visible shadows but is of limited use when such conditions are not available.

Figure 5: Illustration of the results of free space detection run in an indoor environment. Amount of free space (blue area) is in direct correspondence with the proximity of obstacles. Color indicates estimated distance to obstacle (green: d>1m, yellow: 0.5m