Vision-based Real-time Road Detection in Urban Traffic Jianye Lu*, Ming Yang, Hong Wang, Bo Zhang State Key Laboratory of Intelligent Technology and Systems, Tsinghua University, CHINA ABSTRACT Road detection is the major task of autonomous vehicle guidance. We notice that feature lines, which are parallel to the road boundaries, are reliable cues for road detection in urban traffic. Therefore we present a real-time method that extracts the most likely road model using a set of feature-line-pairs (FLPs). Unlike the traditional methods that extract a single line, we extract the feature lines in pairs. Working with a linearly parameterized road model, FLP appears some geometrical consistency, which allows us to detect each of them with a Kalman filter tracking scheme. Since each FLP determines a road model, we apply regression diagnostics technique to robustly estimate the parameters of the whole road model from all FLPs. Another Kalman filter is used to track road model from frame to frame to provide a more precise and more robust detection result. Experimental results in urban traffic demonstrate real-time processing ability and high robustness. Keywords: Autonomous vehicle, lateral visual guidance, road detection, feature-line-pair, Kalman filter

1. INTRODUCTION Autonomous vehicle guidance has been a hot research area in the past 20 years [1][2]. Among the complex and challenging tasks that have received the most attention, road following, which is composed of road detection and obstacle detection, is the most important one. Recently, due to the low cost of camera and well-developed algorithm in computer vision, visual guidance for autonomous vehicle has been highlighted research field, which focuses on machine vision techniques that detect particular features in images of the road ahead of the vehicle, and determine the desired vehicle position with respect to the road boundary based on these features. In the past decades, several road following systems were proposed and demonstrated successfully. Most of them aimed at lateral vehicle guidance on highway or other well-structured road. GOLD (Generic Obstacle and Lane Detection) system reduced road detection to the localization of specific structured features painted on the road surface, such as lane markings [3]. RALPH system at CMU introduced the planar road assumption and tracked the parallel lines on the road, performing robust road following despite degradation of lane markings [4]. PVR III system at Pohang used similar method to implement road detection task [5]. Some other systems extended the detection of lane markings to the detection of road boundaries, typically using gradient operators. All these systems usually have very fast processing speeds and are very well suited for structured roads with good conditions. Although lateral visual guidance for automatic vehicle on highway has been thoroughly explored, few attention has been put on urban traffic, which is characterized by degradation of lane markings, random road geometry, tight curves and “weak” traffic participants like bicycles and pedestrians. It is obvious that such traffic scene becomes very complex and it is much more difficult to reliably interpret. SCARF and UNSCARF systems at CMU have been built to deal with complicated scenes. They have achieved impressive accuracy and robustness on fairly unstructured roads with shadows, leaves lying on the road and lighting changes [6]. But they are too slow to meet the real-time requirement, which is important in urban traffic.

*

[email protected]; phone +86-10-62782266; http://www.lits.tsinghua.edu.cn/lujianye/; State Key Laboratory of Intelligent Technology and Systems, Tsinghua University, Beijing 100084, CHINA

Usually there are many lines parallel to the road boundaries in the urban traffic scene, such as lane markings and vehicle tracks. Such lines, called feature lines in this paper, make up of a great part of all the edge lines in the image and demonstrate some geometrical consistency as a whole. In this paper, we propose a real-time road detection method based on FLPs in urban traffic. Unlike the traditional methods that extract a single line, we extract the feature lines in pairs with global criterions. Working with a linearly parameterized road model, we can detect FLP with a Kalman filter tracking scheme. Since each FLP determines a road model, we apply regression diagnostics technique to robustly estimate the parameters of the whole road model from all FLPs. Another Kalman filter is used to track road model from frame to frame to provide a more precise and more robust detection result. Experiments in urban traffic on THMR-V (Tsinghua Mobile Robot V), an autonomous vehicle developed by Tsinghua University, demonstrate real-time processing ability and high robustness of our method. The rest of this paper is organized as follow. Firstly, several commonly used road models along with three reasonable assumptions are introduced in section 2. Secondly, we give the definition of FLP and a fast, robust FLP detection algorithm with Kalman filter in section 3. Thirdly, we introduce the robust estimation of the road model parameters on the basis of FLP and road model tracking in section 4. Then we present some experimental results and detailed analysis in section 5. Finally, we end this paper with some conclusions and description of future work in section 6.

2. ROAD MODEL A complete description of the road geometry in the image can be complex since the road may vary in width and curvature. The more parameters used in the road model, the greater the chance of error in estimating those parameters and the more computation required. Some road models have been proposed, such as clothoid curve, polynomial curve with low order, parabola, and the simplest triangular model [7]. They perform differently in processing time and detection accuracy. In our method, a linearly parameterized road model is used with the following assumptions: Let (x*, y*, z*) be the point in real world coordinate. 1. The road is planar, i.e. z*=0 with each point on the road plane; 2. The road boundary is approximated by a d-order polynomial: x* =

3.

∑

d i =0

ai y *

i

(1)

where ai is the coefficient of polynomial; The road is equally wide at each point, which may be locally approximated as: * * x 2 = x1 + w * where w* is the constant width between road boundaries.

(2)

According to the assumption 1, transformation between the road plane (x*, y*) and the image plane (x, y) can be described as [8]: x = l x x1* y = ly

y* x*

(3)

where (x, y) denotes the point in the image plane; lx and ly depend only on the camera calibration parameters. We set the origin of the coordinate system to a point on the skyline, which can be computed from prior camera calibration. Combining Eq. (2) and Eq. (3), we may describe one of the road boundaries as follow in the image plane: d x = i =0 bi y 1− i

∑

bi = l y ⋅ ai ⋅ l x

i −1

(4)

According to assumption 3, another boundary may be described in the real world as described in the image as

x2 =

∑

d i =0

l

=Σ

d i=0

*i

*

aiy + w

and may be

bi y 1−i + wy

bi = l y ⋅ ai ⋅ l x w = w* ⋅ l yx

x2*

i −1

(w is also a constant)

(5)

On the basis of the above pair of curves, road model as A = ( bi, w ), 0≤i≤d is defined, which is called linearly parameterized road shape [8]. In this road model, if d=1, the model turns to be the classical triangular model, which is an acceptable approximation in most cases. In other words, we may make an assumption that the road is locally straight. Under the same circumstance, [6] simply describe the road geometry with a four-parameter model as shown in Fig. 1.a, or we may modify it to the one shown in Fig. 1.b.

Vy Vanishing Point

Vx

Vy Vx

Vanishing Point

θ

x1 x2 W Figure 1. Four-parameter road model (a) and (b)

3. DETECTION OF FEATURE-LINE-PAIR 3.1 Feature-line-pair In urban traffic, the road is generally well-constructed, locally flat and equal width, which satisfy the assumptions described in section 2. Furthermore, we notice that feature lines, which are parallel to road boundaries, are reliable cues to detect the road. Therefore, we define feature-line-pair (FLP) as pair of local parallel lines, which are also parallel to the road boundaries. Although feature lines may be split into pieces by occlusion of other vehicles or degradation of shadow, all these FLPs appear to contribute to the same road model. Fig. 2 shows the FLPs in a typical traffic, where line 2 and line 6 are road boundaries. Besides line 2 and line 6, there are many edge lines parallel to them. Each two of them, like line 3 and line 4, line 1 and line 7, compose different FLPs. Obviously each FLP is a part of a triangle, which corresponds to a road model described in Fig. 1. We may easily prove that all triangles corresponding to an FLP share the same vertex, which is called vanishing point. Owing to robust FLP detection introduced below, we can apply simple and fast edge detector to the image before FLP detection. Once to all the FLPs are detected, we can reconstruct the whole road model easily.

Figure 2. FLPs on the road

3.2 Feature-line-pair detection with Kalman filter Traditionally, we can detect the FLP by local connectivity or correlation tracking method [9]. When we work with a specific road model described in section 2, it is possible to detect FLP with the Kalman filter. First, we define each edge point pair (EPP) of an FLP as (width, xL), where width denotes the difference between left point and right point, and xL denotes the x-location of the left point in the image. Combining the Eq. (4) and Eq. (5), we get (width, xL,) = (wy, Σdi=0 bi y1-i ). We may search EPP row by row, beginning from the bottom of the image. As the y decrease, i.e. we search for the upper row, width is expected to decrease by a constant w. Since width linearly depends on y, we may track it with a normal Kalman Filter, which will give out more accurate detection result with high speed. If we let d=1, then xL=b0· y +b1 and it also linearly depends on y, which allow us to track both width and xL within an integrated Kalman filter. Thus we work out an effective and robust method to detect all of the FLPs.

If we simply take d=1, we can track all the EPP with one integrated Kalman filter. Let the measurement vector zk=[width, xL]T. The processing state vector can be described as a four-dimension vector xk=[width, xL, w, vx]T, where w and vx represent the “velocity” of width and xL, respectively. All above variables are define in the (width, xL) coordinates. The processing equation and measurement equation of our Kalman filter can be described as  xk = Axk −1 + wk −1 (6)   z k = Hxk + vk where the subscript k denotes the time instance, A is matrix of linear dynamics system, H is matrix of linear measurement. For the EPP tracking in our application, we may define A and H as 1 0 1 0 1 0

  0 1 0 1  0 1 (7) A= H = 0 0 1 0 0 0     0 0 0 1  0 0 where time interval is assume to be 1. In Eq. (7), the random variables w and v represent the process and measurement noise respectively. They are assumed to be independent of each other, white, and with normal probability distributions p ( w) ~ N (0, Q) (8) p (v) ~ N (0, R ) In practice, the process noise covariance Q and measurement noise covariance R matrices are constant, and they can be measured with some off-line sample measurement.

Although Kalman filter can give real-time performance and reduces computation greatly, it has its own defects. Kalman filter provides a recursive solution of the least-square method, and it is not a robust estimator. It is incapable of detecting and rejecting the outliers, which may cause collapse of tracking. Besides, Kalman filter records not the data ever measured but only the combination states at time k-1, which means that the final detection result is sensitive to the order of measurement. Sometimes such properties will worsen the accuracy of the detection result. To alleviate these problems, we define a measurement describing the reliability of current Kalman tracking. The reliability of the next measurement appended to current Kalman filter is the ratio between the correction increment SB and the prediction increment SA , as shown in Fig. 3. R = ( SS BA ) 2 Rk* = l ⋅ Rk −1 + (1 − l ) ⋅ Rk

where

(02. However, we find in practical experiments that complex road model seems so unstable that even small noises in the edge image will greatly affect the detection result. Currently we are working on such problems.

(a) vx tracking in images sequence

(b) vy tracking in images sequence

Figure 6. Tracking of measurements of the four-parameter road model

Our method has also been tested with several images sequences. Fig. 6 shows the measurements of the road model parameters in one of the tests. Owing to the Kalman filter tracking, some errors are corrected and we get quite stable detection results. Our method also shows real-time performance in all the experiments due to the use of Kalman filter tracking. For the sequence with 128x128 images in Fig. 10, the whole processing takes 4 ms to detect the edge pixels, 80 ms to detect all the FLPs, 11 ms to estimate the road model parameters, and no more than 1 ms to determine the exact road model with Kalman filter tracking. Time costs add up to no more than 96 ms, which satisfied the real-time requirement in urban traffic.

6. CONCLUSIONS AND FUTURE WORK A robust detection and tracking of road shape via on-broad camera becomes more and more important for autonomous vehicle guidance. For lateral vehicle guidance, road boundaries detection must at least provide at a good rate estimates of the relative orientation and of the lateral position of the vehicle with respect to the road [8]. In this paper we proposes a real-time road detection method in urban traffic. Extracting feature-line-pair instead of single feature line makes gives more robust detection result despite degradation and occlusion of lane markings. Working with a linearly parameterized road model, FLPs detection with EPP tracking may achieve effectively. Regression diagnostics technique is used to robustly estimate road parameters from all FLPs, which gives us more accurate and more robust road shape. Experiments on THMR-V demonstrate robust and real-time performance. Currently, we are working on optimizing this method, and extending the technique for detecting more complex road model in more generic urban traffic scene. We will begin more research on robust tracking in image sequence, and improve the real-time performance and robustness in dense traffic.

ACKNOWLEDGEMENTS This research was supported in part by Chinese High Technology Development Program and Portugal-China Science and Technology Cooperation Project. We wish to thank Bin Dong, Qian Yu for their valuable help on this paper.

REFERENCES 1.

R. Chapuis, A. Potelle, J.L. Brame, et al, “Real-time vehicle trajectory supervision on the highway,” International Journal of Robotics Research, 14, pp.531-542, 1995. 2. C. Thorpe, M. Herbert, T. Kanade, et al, “Vision and navigation for the Carnegie-mellon navlab,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 10, pp. 361-372, 1988. 3. M. Bertozzi, A. Broggi, “GOLD: A parallel real-time stereo vision system for generic obstacle and lane detection”, IEEE Transaction on Image Processing, 7, pp. 62-81, 1998. 4. D. Pomerleau, “RALPH: rapidly adapting lateral position handler,” IEEE Symposium on Intelligent Vehicles, pp. 506-511, 1995. 5. K.I. Kim, S.Y. Oh, S.W. Kim, et al, “An autonomous land vehicle PRV III,” in proceedings of the IEEE Intelligent Vehicles Symposium, pp. 159-164, 1996. 6. C.E. Thorpe, Vision and navigation - the Carnegie-Mellon Navlab, Kluwer Academic Publishers, 1990. 7. M. Yang, J.Y. Lu, H. Wang, et al, “Vision based road following,” Chinese Journal of Pattern Recognition and Artificial Intelligence, 14, pp. 186-193, 2001. (In Chinese) 8. F. Guichard, J. -P. Tarel, “Curve finder combining perceptual grouping and a kalman like fitting,” in International Conference on Computer Vision, pp. 1003-1008, 1999. 9. Z. Li, et al, “Dynamic Image Analysis,” China National Defence Industry Press, 1999. 10. Z. Zhang, “Parameter estimation techniques: a tutorial with application to conic fitting,” in Image and Vision Computing Journal, 15, pp. 59-76, 1997.