Automatic tuning via Kriging-based optimization of methods for fault detection and isolation Julien Marzat, Eric Walter, Hélène Piet-Lahanier, Frédéric Damongeot Conference on Control and Fault Tolerant Systems Nice , France, October 6-8 2010
SYSTOL 2010 - J.Marzat - 07/10/2010 - 1/15
Outline Problem formulation Tuning methodology Kriging Efficient Global Optimization Case study Results Summary and future work
SYSTOL 2010 - J.Marzat - 07/10/2010 - 2/15
Problem formulation Hyperparameters of fault diagnosis methods Observer gains, thresholds, size of expected change, width of a sliding window... Tuning of Hyperparameters Required to compare fault detection methods Involves costly simulations of a test case Global minimization of performance indices to find hyperparameters
SYSTOL 2010 - J.Marzat - 07/10/2010 - 3/15
Problem formulation Application proposed Abrupt change in the mean of Gaussian and uniform signals 6 candidate methods to be tuned, involving 1 to 4 hyperparameters Tuning viewed as a Computer Experiment Kriging, surrogate approximation of the complex simulation Efficient Global Optimization, iterative search for the global optimizer based on the Kriging prediction Starting point Performance index y (x) already computed for an initial sampling of hyperparameter vectors Xn = [x1 , ..., xn ], x ∈ X ⊂ Rd
SYSTOL 2010 - J.Marzat - 07/10/2010 - 4/15
Kriging y (·) modeled as a Gaussian process Y (x) = f T b + Z (x) where f parametric prior, b to be estimated from the available data Z (·) zero-mean Gaussian process with covariance k (Z (x), Z (x + h)) = σ 2 R (h) o n P d p Chosen correlation function R (h) = exp − k=1 |hk /θk | k Empirical Kriging: θk , σ 2 estimated by maximum likelihood What Kriging provides b (x), best linear unbiased prediction of y (·) at any x ∈ X Y Variance of the prediction error σ b2 (x)
SYSTOL 2010 - J.Marzat - 07/10/2010 - 5/15
Kriging illustration
SYSTOL 2010 - J.Marzat - 07/10/2010 - 6/15
Efficient Global Optimization (EGO) algorithm Objective: find iteratively the minimum of y (·) 1
2 3 4
Choose an initial sampling and compute the value of y (·) at those points Find the empirical minimum in the available data points, ymin Fit a Kriging predictor on those data points Find a new point of interest to evaluate y (·) by maximizing Expected Improvement, given by EI(x) = σ b (x) [uΦ (u) + φ (u)] where
u=
5
b (x) ymin − Y σ b (x)
Go to step 2 until EI reaches a threshold or sampling budget is exhausted
SYSTOL 2010 - J.Marzat - 07/10/2010 - 7/15
Illustration of EGO
Iteration 1
Iteration 2
Iteration 3 SYSTOL 2010 - J.Marzat - 07/10/2010 - 8/15
Test cases Change from 0 to 1 in the mean at t=500 of a Gaussian signal with unit variance and a uniform signal in [−2; 2]
Cost function: weighted sum of Detection delay, false-detection rate, non-detection rate
SYSTOL 2010 - J.Marzat - 07/10/2010 - 9/15
Methods to be tuned
1
3–sigma fixed threshold
2
Student test on a sliding window
3
GLR (Generalized Likelihood Ratio Test)
4
SPRT (Sequential Probability Ratio Test)
5
CUSUM (Cumulative Sum)
6
RSS (Randomised SubSampling)
3–Sigma ν
Hyperparameters to be tuned Student GLR SPRT CUSUM N N, λ N, µ1 , α, β δ, λ
SYSTOL 2010 - J.Marzat - 07/10/2010 - 10/15
RSS N, q, M
Results - Decision functions for Gaussian test case
SYSTOL 2010 - J.Marzat - 07/10/2010 - 11/15
Results - Decision functions for uniform test case
SYSTOL 2010 - J.Marzat - 07/10/2010 - 12/15
Results - exploration of hyperparameter spaces
SYSTOL 2010 - J.Marzat - 07/10/2010 - 13/15
A sample of numerical results Gaussian test-case 3-sigma
Student
GLR
SPRT
CUSUM
RSS
Ranking
6
4
5
3
1
2
Best cost c1
0.7573
0.0559
0.0699
0.0359
0.024
0.0339
Detection delay
501
28
35
18
12
17
False alarm
0.2124
0
0
0
0
0
Non-detection
0.5449
0.0559
0.0699
0.0359
0.024
0.0339
Uniform test-case 3-sigma
Student
GLR
SPRT
CUSUM
RSS
Ranking
6
5
4
3
1
2
Best cost c1
0.7585
0.1141
0.0978
0.0359
0.01
0.0339
Detection delay
501
37
47
285
5
30
False alarm
0.002
0.0741
0.004
0
0
0
Non-detection
0.7565
0.0399
0.0938
0.0359
0.01
0.0339
SYSTOL 2010 - J.Marzat - 07/10/2010 - 14/15
Summary and future work Summary Automatic tuning with tools from Computer Experiments Kriging provides a surrogate model Iterative global optimization based on Expected Improvement Successful tuning and comparison of 6 change-detection methods Few runs of the simulation (less than 100) Current developments Tuning of complete FDI schemes involving residual generator and residual evaluation for a dynamical system Taking into account environmental variables (noise level, size of faults, model uncertainty...)
SYSTOL 2010 - J.Marzat - 07/10/2010 - 15/15
