Perspectives on (multi-) UAV Operations Autonomy & Safety Issues
Patrick Fabiani
[email protected]
Toulouse
IAV2007 3-5th September 2007
Uninhabited Air Systems : what for ? ≡
Exploration / Inspection / Surveillance missions by uninhabited aircraft ? ∨
dirty, dull or dangerous missions •
forest or bush fires, chemical or nuclear accident areas, power cables or pipe line, borders, maritime routes, flooded regions, search and rescue . . .
∨
≡
MAVs allow to limit the risks for inspection missions
From existing UA Vehicles … …
To “One UA System design for each purpose”: chemicals spraying over tea or rice fields
Commercial success for the RC version ! Page 2
Toulouse
MAV 2007
17th-21st September 2007
Uninhabited Air Vehicles : what perspective ? ≡
Feasibility is explored in Research Projects ∨
Search & Rescue : International Aerial Robotics Competition (IARC 1998-2000) •
∨
Fire Fighting : EU “COMETS” project •
∨
Technische Universität Berlin MARVIN
coordinated flight of heterogeneous unmanned air vehicles autonomous or remotely controlled
Ground traffic surveillance and supervision : Univ. Linköping “WITAS” project •
autonomous takeoff, navigation, landing and vehicle tracking, planning, situation assessment, voice control
Page 3
∨
Insertion in the General Air Traffic : EU “USICO”, EDA “Sense & Avoid”,
∨
Configurations for Civil Applications : EU “CAPECON”
∨
Urban exploration and reconnaissance : New Challenge for the IARC ! Toulouse
MAV 2007
17th-21st September 2007
Autonomy for Uninhabited Vehicles REDERMOR (DGA/GESMA)
CEVM (SPNuM / EADS)
2006
2006
New missions ?
? Page 4
?
? Toulouse
?
MAV 2007
17th-21st September 2007
Uninhabited Air Systems : what perspective ? ≡
Feasibility can only be fully demonstrated in Flight
≡
Tight integration constraints
≡
First achievable on bigger demonstrators … … Then miniaturized on board smaller UAVs, and MAVs
≡
Increasing the difficulties of the mission … … increasing the autonomy challenges
≡
Page 5
Autonomy ?
Toulouse
MAV 2007
17th-21st September 2007
Autonomy for Underwater Vehicles Full autonomy - 22 mars 2006 REDERMOR Autonomous Underwater Vehicle « NIVAS »
Data flow
Page 6
Toulouse
MAV 2007
17th-21st September 2007
Autonomy of Orbital Systems ONERA - CNES : ASO program AGATA project : generic multi-mission autonomy architecture
Distance Autonomous sub-systems: (attitude, orbit, thermics,.)
Advanced management (resources allocation, planning, etc..)
limited flow data link no real-time constraint mission control center Page 7
Toulouse
mission control center MAV 2007
17th-21st September 2007
Autonomy of Multi-UAV Systems
11
22
77
88
99
! !
55
10 10 66
44
Cooperative Decision Making : ARTEMIS
"adjustable" "adjustable" Wpt Wpt
20 20 !# !#
24 24
23 23
!!""## $$!! %% "& && '' ('" "& ('" #) #" )""* "*
22 22 # #
$ % # $ % #
$
– anytime decision making – cooperation of distributed agents – decision aid and man-machine
Page 8
Toulouse
MAV 2007
17th-21st September 2007
16a 16a 16b 16b
12 12
" "
% %
– planning algorithms & models
!
15 15 17 17
14 14 19 19 $
Complexity
interaction
18 18 ' ( ' ( ' '
22 11
Distributed anytime decision for a team of Combat UAVs (demonstrated in simulation with Dassault Aviation for DGA)
11 11!
10bis 10bis 33
13 13
& &
+, +,
Autonomy for Unmanned Air Systems ? What For ? ≡
Not a goal in itself ∨
≡
≡
Page 9
Complexity of remote operations
Operators workload and situation awareness ∨
management of information and data flows in complex networks or trafic
∨
distance and delays
∨
multiple UAV operations
Efficiency, Safety and Dependability ∨
reduced operator work load
∨
increase the control efficiency for flight & sensors
∨
compensate for data link failures
Toulouse
MAV 2007
17th-21st September 2007
Lessons learnt about uninhabited systems
≡
Page 10
Safety
What kind of autonomy ? ∨
the autonomous aircraft must always act within the limits
∨
the autonomous aircraft must locally react to event
Toulouse
MAV 2007
17th-21st September 2007
Lessons learnt about uninhabited systems ≡
Crucial : mobility and safety
≡
Operator in the loop
≡
Reactivity (cooperative environment)
≡
Simulated ship decking on a virtual moving point •
Operational autonomy
Page 11
Toulouse
distantly designated by a GPS-IMU equipped pointing device (moving on the SIREHNA platform)
MAV 2007
17th-21st September 2007
Lessons learnt about uninhabited systems Canadair CL289 1995 NATO - DGA/DCN
≡
Decking on a ship at sea is hard
≡
More efficiency and safety is required
≡
The percentage of aborted attempts is still not reasonable for operational use: complementary sensors are necessary for greater reliability & availability (GPS)
≡
Simulated motion of the ship at swell is easier than real swell at sea.
≡
The helicopter limited maneuvering capabilities demand an improved strategy
≡
Other example: “sense & avoid” real time mobile obstacle avoidance mainly depends on the detection (“sense”) capability.
Page 12
Toulouse
MAV 2007
17th-21st September 2007
Lessons learnt about uninhabited systems Autonomous navigation can be authorized on predefined flight plans
≡
DGA/SPMT - TSI Vigilant F2000 (35kg) « out of sight » Autonomous Flight Authorization by French Civil Aviation Authorities around Revel airfield since 1997
Safe & Verifiable Flight Control & Mission Supervision Architecture
≡
is very important crucial ! FUJI (300kg) TSI / VigilantF5000 (2000)
Page 13
Toulouse
MAV 2007
ONERA ReSSAC (Yamaha RMaX) “line of sight” Autonomous Flight Authorization 17th-21st September 2007
ReSSAC project : decision & control architecture #
$
Ground
" Inputs should never affect configuration
! " Page 14
Toulouse
MAV 2007
17th-21st September 2007
Flies !
Autonomy for Uninhabited Air Systems ≡
New missions ?
Decisional Autonomy ? Decision making within a (verifiable) domain for achieving a (common) goal
in spite of unexpected events or situations … Examples: • landing on unprepared terrain (hostile area or emergency) • acting in a partially known environment (search/exploration) • acting in a dynamic environment (adaptation)
Page 15
Toulouse
MAV 2007
17th-21st September 2007
Perception : Binocular Stereovision ≡
Stereovision processing of pairs of images from two cameras ∨
Pixel matching and triangulation
∨
Detection of obstacle zones (elevation)
∨
Projection back on the terrain geometry
Left
Page 16
Right
Toulouse
MAV 2007
17th-21st September 2007
2 Cameras
Perception: scanning with a laser rangefinder ≡
Page 17
Processing of series of range spots
Toulouse
MAV 2007
17th-21st September 2007
Perception by Scanning with a laser range finder : flies ! ≡
Page 18
Processing of series of range spots
Toulouse
MAV 2007
17th-21st September 2007
Stereovision from motion : principle ≡
Stereovision processing of pairs of images ∨
Pixel matching and triangulation
∨
Detection of obstacle zones (elevation)
∨
Projection back on the terrain geometry
B Camera orientation & trajectory
Camera orientation & trajectory
H
Page 19
Toulouse
MAV 2007
17th-21st September 2007
Stereovision from motion : pixels matching ≡
RANSAC (random sampling based) pixels matching using an homographic model
(hyp.: scene is mostly planar, obstacles rejected as “outliers”, non-linear camera model)
residual motions reveal obstacles (or motions of objects in the image) Page 20
Toulouse
MAV 2007
17th-21st September 2007
Stereovision from motion : Dense approach ≡
Dense Stereovision + Optic flow ∨
speed norm is related to elevation
Esperce80, images 0-19 Page 21
Toulouse
MAV 2007
Norm of the flow, images 0-19 17th-21st September 2007
Stereovision from motion : Dense approach ≡
Dense Monocular StereoVision
Camera trajectory and orientation
~ a pair of images every 800ms
Page 22
Toulouse
MAV 2007
17th-21st September 2007
Stereovision from motion : Sparse approach ≡
Sparse stereovision from motion
selection and matching of points of interest (Harris filter) with adapted density ≡
Sparse monocular stereovision from motion: but sparse elevation map !
Page 23
Toulouse
MAV 2007
17th-21st September 2007
Stereovision flies ! ≡
US Army ARMDEC NASA Ames Research Center
Left
ONERA Esperce Air Field
Right
• Height map from stereo • Laser Scanner
Page 24
≡
Toulouse
MAV 2007
• Height map from motion • Monocular stereo from motion
17th-21st September 2007
Stereovision from motion : Flies !
Page 25
∨
H=10m, speed ~1m/s
∨
Dense approach
∨
vehicle detection (movie is not in real time)
Toulouse
MAV 2007
17th-21st September 2007
Stereovision from motion : Flies !
Page 26
∨
H=10m, speed ~1m/s
∨
Dense approach
∨
cable detection ! (movie is not in real time)
Toulouse
MAV 2007
17th-21st September 2007
Stereovision from motion : Flies ! ≡
Page 27
Stereo from motion (time according to the density of processed points) ∨
Sparse : EVA 1.0 (PIP9 1GHz) 4 sec/frame
∨
Dense : EVA 2.0 (PIP9 1GHz) 10 sec/frame
∨
further work on the coupling of aircraft motion and stereovision processing
∨
( & a 2GHz processor)
Toulouse
MAV 2007
17th-21st September 2007
Deliberative Planning : Autonomous Path Planning
Page 28
Toulouse
MAV 2007
≡
Numerical terrain Model
≡
Crests & Valleys Computation
≡
Model of the piloted aircraft
≡
Feasible trajectories generation
→
Optimal & low altitude itineraries
17th-21st September 2007
Acting and Deciding in ill-known environment 0. Mission preparation 1. Coarse level exploration & characterization obstacles & sub-zones (probabilities of “landable”) re-planning to search for more information
2. Second level exploration and selection of candidate landing areas 3. Landing site characterization and landing if appropriate, otherwise go back to 2. Page 29
Toulouse
MAV 2007
17th-21st September 2007
Autonomous Planning under Uncertainty : MDP flies ! (Dynamic Bayesian Networks) • Structured Markov Decision Processes → state and action variables
-
!-
• Input: mission re-planning problem → mission variables
.-
-
*
→ map, waypoints, itinerary graph → navigation variables and flight time/energy variables → probabilities of transition
Page 30
Toulouse
MAV 2007
17th-21st September 2007
%
&&&&&&&' ' ' &&' %
()
Autonomous Planning under Uncertainty : MDP flies !
#
$
• Embedded re-planning system → anytime algorithm → multi-thread implementation → flight tested
• Output : “Optimal” reactive strategy → conditional to perception ? → which sub-zone to explore ?
! "
Page 31
Toulouse
→ when to abort mission and go home ?
MAV 2007
17th-21st September 2007
Achievements Flight demonstrated between July 2006 - July 2007
- autonomous flight, guidance & navigation, - autonomous take-off and landing, - 4D dynamic rendez-vous (virtual ship decking) - perception: characterization of sub-zones of the initial search zone for further exploration - on-line planning of the exploration of the subzones according to probabilities of “landability” - landing site characterization of each possible subzone for search of a “landable” zone - autonomous landing : spot selection and landing (supervised by the security operator)
Page 32
Toulouse
MAV 2007
17th-21st September 2007
Lessons learnt about uninhabited systems ≡
A real Uninhabited Air Vehicle is never alone !
≡
Flying UAVs in the real world is an adventure
≡
The real world is an ill-known, unprepared and changing environment
≡
Decisional Autonomy can help but Auto-Adaptive systems are, for the moment, not welcome for certification :
Page 33
∨
the deterministic safe panic button is absolutely always required !
∨
developing an efficient, robust, verifiable, adaptive fail safe system is hard
Toulouse
MAV 2007
17th-21st September 2007
Two steps towards certification (among others) #
$
"
Operator must be correctly trained
« Auto - Adaptation » means that inputs may affect the configuration ! "
Page 34
Toulouse
MAV 2007
17th-21st September 2007
Perspectives : ≡
Mobility in non- cooperative environments ∨ sensor (vision) based navigation among obstacles ∨ robustness, dependability, availability, safety, independence from GPS, ...
≡
Cooperative embedded autonomy
≡
∨
autonomous rotorcraft and ground robots in cooperation for a common mission
∨
coordination functions also necessary (formation flight, visual servoing, swarm …)
Dependability of embedded systems : safety, airworthiness, ∨
how to validate and prove the dependability / safety of a highly reconfigurable control architecture including mission re-planning
∨
how to prove safety, reliability and efficiency? how to certify ?
AeROS Lab.(LAAS-CNRS+ONERA) + ISAE Page 35
Toulouse
MAV 2007
17th-21st September 2007
