Lane-Departure Detection and Control System for Commercial Vehicles Gábor Kovács, József Bokor, László Palkovics, László Gianone, Ákos Semsey and Péter Széll

Abstract--The demand of the society on the safety of the transport process has been significantly increased with the growing number of vehicles on roads. The application of intelligent systems in the chassis offers an ultimate solution to improve the commercial vehicle as thus the traffic safety. The paper presents a system for detection and prevention of the unintended lane-departure. It is realised by the integration of a drive stability and a computer vision system. The possible lane-departure is calculated on the basis of the recognised lane geometry and the measured vehicle’s state Then the path of the vehicle is controlled by unilateral braking if lane-departure is predicted. Index Terms-- vehicles, automotive control, computer vision

I. INTRODUCTION The growing volume of the traffic all around the world requires higher and higher levels of the traffic safety. However, the transportation infrastructure cannot keep up with the rising number of vehicles. Thus the traffic flow should be controlled in a way which provides an enhancement of the traffic safety and increases the efficiency of the transportation, i.e. increases the traffic density. There is a contradiction between these two requirements, because increasing traffic density results in growing probability of traffic accidents. This contradiction cannot be relieved, but the probability of accidents can be reduced in a way of giving intelligence both to the infrastructure and to the vehicle itself, making the information flow between the environment and the vehicle possible. The traffic flow control by traffic signals cannot solve this problem completely since the reaction of the human driver can be incorrect, slow and it depends on the nature and mood of the driver. These deficiencies of the human driver can be compensated by the application of intelligent vehicle systems which close the vehicledriver-environment loop in a faster and more correct way. The paper presents a computer vision based experimental system for the detection and prevention of the unintended lane-departure. Recently the automobile industry spends a lot of effort on the research and development of such intelligent vehicle systems, see e.g. [3], [5], [10]. The presented system realises a cooperation of the DSC (Drive Stability Control) system and G. Kovács and J. Bokor are with the Computer and Automation Research Institute of the Hungarian Academy of Sciences. L. Palkovics, L. Gianone, Á. Semsey and P. Széll are with the Knorr-Bremse Brake Systems Ltd., R&D Institute, Budapest, Hungary.

the computer vision system. The basic function of the DSC system is to assist the driver to maintain the desired path and it intervenes when the actual path of the vehicle differs from the desired one. For a detailed description of drive stability systems, see e.g. [1], [2], [6], [7], [8], [9]. The integrated system is able to recognise that an unintended lane-departure is happening, it alerts the driver and corrects the trajectory of the vehicle by activating the DSC system. II. DESCRIPTION OF THE CONTROL LOOP Fig. 1 shows the control loops of the system. The vehicle is equipped with a digital camera for sensing the lane geometry. An estimation for a possible lane-departure is calculated from the processed images and from the prediction of the vehicle’s motion. A decision logic is built to activate the warning signal and the intervention on the basis of the driver’s activity and of the status of the DSC system to maintain safety. The primary control role of the driver is always kept and lateral stabilisation functions of the DSC have priority over the lane-keeping intervention. There are reaction times for warning ( Tw ) and for intervening ( Ti ) defined. If there is no driver’s activity and a lane-departure is predicted some period of time ahead less than the reaction time for the warning, an audible warning signal is given for paying the attention of the driver. If the estimated period of time for lanedeparture is less than the reaction time for intervention, then intervention to the vehicle’s motion is applied using  the DSC system. The necessary required yaw rate Ψ r of the vehicle is calculated which should be followed by the trajectory of the vehicle’s motion that the vehicle remains in the lane. The DSC system applies a torque to the vehicle by unilateral application of the brakes to modify the vehicle’s motion according to its desired yaw rate. The lane-departure detection system consists of a PC with INTEL PENTIUM 133 MHz processor, a MATROX METEOR PCI frame grabber card and a COMERSON TC230CCD camera. The computer for image processing and lane-departure detection is connected to the data acquisition computer via a CAN serial communication interface to receive measurements needed to predict the vehicle’s path. The on-board computers are connected via the CAN line to an ABS/ASR ECU prepared specially for receiving external valve commands. The test vehicle is equipped with some additional on-board sensors for the DSC function for measuring the yaw rate, the steering angle, the lateral acceleration, and the brake chamber pressures. Wheel speed signals are obtained from the ABS system. The required pressure for autonomous

1998 IEEE International Conference on Intelligent Vehicles

46

driver

steering, brake,

road scene

warning lanerecognition

 Ψ r

comparison

trajectory prediction

decision

DSC system

vehicle

Tw Ti states of motion

view from the vehicle Fig. 1. Control loops of the lane-departure prevention system

unilateral braking is produced by properly controlled ASR valves. It is necessary to emphasize that the designed system is experimental, current commercial vehicle electronics is not yet capable of implementing tasks with such high computational complexity like image processing.

α est = arg max δp(α k ) αk

1

This method for estimation of the road curvature is independent of the type of lane boundaries. They can be formed with solid or broken line lane marks, the lane boundary can be paved or it can be a simple transition from asphalt to grass (or gravel).

III. LANE GEOMETRY RECOGNITION In order to detect the lane-departure, it is necessary to produce a robust estimation of the road geometry in front of the vehicle. Three parameters have to be determined: the road curvature, the width of the lane, and the vehicle’s lateral offset from the lane centre. The detection algorithm is based on the optimisation of two functionals. Step 1: Estimation of the road curvature. The radius of the road curvature is relatively large on public roads and highways. Thus the road section in front of the vehicle can be approximated with a straight line. The detection system determines the angle α between this straight line and the front face of the vehicle. Consider Fig. 2. The perspective view of the roadway can be transformed to a top view picture with parallel lane boundaries using equations of the perspective transformation. Then a 2 N + 1 -dimensional array of the examined discretised angle range is defined: [−α N ,  ,0,  , α N ] . An intensity profile vector p(α k ) is assigned to each angle α k . This vector contains the sums of the grey-scales intensities in direction under angle α k starting from the pixels of the bottom line of the top view road image. Next the vector corresponding to α should be selected. An optimisation problem is formulated for this task. Another set of vectors δp(α k ) containing absolute values of the differences between elements of each intensity profile vector p(α k )

is formed. Then the angle α is estimated as:

Fig. 2. Input image and recognition of the lane geometry

1998 IEEE International Conference on Intelligent Vehicles

47

The curve on the left side in the processed image of Fig. 2 shows the estimated angle as a function of the time. The highest bar in the lower of the two bar graphs above the trapezoid of the top view image shows the estimated direction α est , meanwhile the upper bar graph shows the corresponding difference vector δp(α est ) .

Step 2: Estimation of the lane width and the vehicle’s lateral offset from the lane centre. To solve these problems another optimisation method is formulated which uses the result of the Deriche’s edge detection operator applied to the input image. This powerful edge detection operator was derived with the optimisation of three functionals characterising the noise reduction, the localisation, and the uniqueness properties [4]. The method for determining the width of the lane and the vehicle’s relative offset from the lane centre is as follows. A pair of parallel lines with angle determined in Step 1 are moved in the top view trapezoidal image (See bottom sketch on Fig. 2). The distance of the parallel lines is varying. The best fit to the significant edges determines the needed parameters. This method can be formalised as follows. Let w ∈[w min , w max ] denotes the width of the lane

and o ∈[− w 2 , w 2] the vehicle’s lateral offset from the

centre of the lane. Let v(α est , d , i ) be a vector of the length d under the estimated angle α est starting from a bottom line pixel of the top view image (formed from the result of the Deriche’s edge detection operator applied to the original image) shifted from the centre pixel by i points to the right. v j (.) denotes its j th element. The

parameters w and o can be estimated using the following formulae w oest = iopt − est 2

[

w

est

]

, iopt = arg

d

max

∑v (

w ∈[ wmin , wmax ] j =1 i ∈[ 0 ,w ]

j

α

est

, d , i − w) v j (

α

est

, d ,i)

All the estimated parameters have been filtered with a 2nd-order Butterworth filter to achieve better performance. The second and the third curves from the left on Fig. 2 show the vehicle’s lateral offset from the lane centre and the estimated lane width. The presented optimisation procedure can be easily generalised to higher order curves approximating the lane geometry in front of the vehicle. IV. CONTROL DESIGN The resulted parameters of the lane-departure prediction determine the desired yaw rate for the vehicle. The selected strategy for determining the desired vehicle’s path is to force the vehicle to follow an arc the tangent of which is the lane border (or a line lying some distance from the border towards the centre of the lane), and assume that both the angle of lane-departure and the steering angle are small. Then the desired yaw rate is obtained as:

γ 2Tc where γ denotes the angle of crossing the boundary of  = Ψ r

the lane, and Tc is the estimated time to lane crossing (which is equivalent to the estimated distance divided by the current forward speed of the vehicle). The required trajectory determined by the desired yaw rate is followed by the controller of the DSC system. Generally the DSC calculates the optimal vehicle responses on the basis of the intention of the driver (and any higher level control systems), and of the measured and/or estimated actual state variables. If the vehicle response differs from the calculated optimal response, DSC actuates the wheel brake on the appropriate side which produces the stabilising torque. Due to the highly non-linear behaviour of the vehicle’s motion in the operation range where DSC may become active, its optimal control algorithm cannot be described in a closed form taking all driving situations into the consideration. Concerning the lane-departure prevention problem, the following assumption is made for modelling: The vehicle’s motion can be described by a single track model which assumes small steering and side slip angles and slowly changing forward speed. Furthermore the effect of the small side slip angle is neglected. The equation of motion is obtained as  = F l − F l − M JΨ 11 2 2 0 Fi = ci βi

βi = (2 − i )δ s + ( − 1)

i

 li Ψ vf

where J denotes the moment of inertia of the vehicle’s body, F1 and F2 denote the side forces at the front, respectively the rear wheels, l1 and l2 are the distances from the front, respectively rear axles to the centre of gravity, ci are the cornering stiffness at the corresponding  the yaw rate, v wheel, δs denotes the steering angle, Ψ f

the forward speed of the vehicle, M 0 denotes the additive torque generated by the DSC. In the case of unilateral application of the rear brakes, M 0 depends approximately linearly from the difference between the pressures of the rear brake chambers of the sides of the vehicle until some saturation limit: M 0 = d 2 cb pd where pd denotes the pressure difference, cb is the brake coefficient, and d 2 is the half of the rear wheel track. Then the state equation is obtained as: x = ax + b1u1 + b2 u2 where 2 + c2 l22 cl dc  , a = − c1l1 , b1 = 1 1 , b2 = − 2 b x=Ψ Jv f J J u1 = δs , u2 = pd This model can be identified from the measured signals using on-line identification method to capture the changes of the working point of the model. The best applicable control design method based on the identified model is now under examination (PD, finite and infinite horizon LQ, Robust LQ, etc.). Note that the assumption made for

1998 IEEE International Conference on Intelligent Vehicles

48

Fig. 3. A sample of roadway test situations

Fig. 4. Results of the lane recognition

modelling is not unrealistic having quasi straight vehicle motion with quasi constant speed (typical conditions during the tired driver most likely falls asleep) and not very slippery road surface. The unstable vehicle motion can however be detected from the on-vehicle sensors, in which case the whole DSC control algorithm can be activated.

generated by the vision system activates the unilateral braking application. In the case of the performed tests presented on Fig. 5 and 6 a pressure proportional to the reference yaw rate is produced in the rear brake chamber on the proper side. Signals of the vehicle state during intervention are depicted on Fig.5. Subsequent pictures of the resulted vehicle’s behaviour are presented on Fig. 6.

V. EXPERIMENTAL RESULTS A set of pictures on Fig. 3 and 4 demonstrates intensive tests of the vision system performed in various road, weather and lighting conditions. The trigger signal

VI. CONCLUSION A system for detection and prevention of the unintended lane-departure has been presented. It is based on the integration of a computer vision system for the lane

1998 IEEE International Conference on Intelligent Vehicles

49

recognition and a drive stability control system. The described image processing algorithm provides a robust estimation of the lane geometry under different environmental circumstances. Experimental results have shown that lane-departure can be avoided by unilateral application of the rear wheel brakes. 12

front left wheel speed [m/s] 10

front right wheel speed [ /]

8

6

pressure of the right rear brake chamber [bar] 4

2

trigger from the vision system

0

yaw rate of the vehicle [0.1/s] -2 60

60.5

61

61.5

62

62.5

63

63.5

Fig. 5. State signals of the vehicle during the intervention VII. ACKNOWLEDGEMENT This research has been supported by the National Committee for Technological Development through grant No. 06249970203, which is gratefully acknowledged. VIII. REFERENCES [1]

Fig. 6 The controlled vehicle’s behaviour

J. Ackermann and T. Bünte, “Yaw Disturbance Attenuation by Robust Decoupling of Car Steering,” Control Engineering Practice, 8, pp. 1131-1136, 1997. [2] V. Alberti and E. Babbel, ”Improved Driving Stability by Active Braking of the Individual Wheels,” Proceedings of AVEC’96, Aachen, Germany, pp. 717-732, June 1996. [3] E.D. Dickmans and N. Müller, “Scene Recognition and Navigation Capabilities for Lane Changes and Turns in VisionBased Vehicle Guidance,” Control Engineering Practice, 4, pp. 589-599, 1996. [4] O. Faugeras, “Three-Dimensional Computer Vision - A Geometric Viewpoint,” The MIT Press, Massachusetts Institute of Technology, 1993. [5] D.J. LeBlanc, G.E. Johnson, P.J.Th. Venhovens, G. Gerber, R. Desonia, R.D. Ervin, Ch.F. Lin, A.G. Ulsoy and T.E. Pilutti, “CAPC: A Road-Departure Prevention System,” IEEE Control Systems, pp. 61-71, December 1996. [6] A. Müller, W. Achenbach, E. Schindler, T. Wohland, F.W. Mohn “Das neue Fahrsicherheitssysteme Electronic Stability Program von Mercedes Benz,” ATZ, 96. Jahrgang, Nr. 11, pp. 656-670, 1994. [7] L. Palkovics, L. Ilosvai and Á. Semsey, “Study on self-steering behaviour of high-speed tractor-semitrailer and its possible control improving the lateral stability,” International Journal of Vehicle Design, Series B, Vol. 1, No. 3, Heavy Vehicle System, pp. 304324, 1994. [8] L. Palkovics, M. El-Gindy, "Design of an Active Unilateral Brake Control System for Five-Axle Tractor-Semitrailer Based on Sensitivity Analysis," Vehicle System Dynamics, Vol. 24, pp. 725758, 1995. [9] T. Pilutti, A.G. Ulsoy and D. Hrovat, “Vehicle Steering Intervention Through Differential Braking,” Proc. of American Control Conference, Seattle, 3, pp.1667-1671, 1995. [10] D. Pomerlau and T. Jochem “Rapidly Adapting Vision for Automated Vehicle Steering,” IEEE Expert, pp. 19-27, April 1996.

1998 IEEE International Conference on Intelligent Vehicles

50