Contributions to multiple testing theory for high dimensional data Etienne Roquain1 1
Laboratory LPMA, Université Pierre et Marie Curie (Paris 6), France
HdR defense, 21-th september 2015
Etienne Roquain
Contributions to multiple testing theory
1 / 37
1 Introduction
Motivation Setting Classical results
2 Adaptive procedures
Adaptive control Adaptive error rate Adaptive p-values Adaptive classification
3 Outlook
Etienne Roquain
Contributions to multiple testing theory
2 / 37
Single testing I Data: X = (X1 , . . . , Xn ) i.i.d. N (µ, 1), µ ∈ R unknown I Null hypothesis H0 : “µ ≤ 0" against alt. H1 : “µ > 0" Pn I Test statistic: T (X ) = n−1/2 i=1 Xi , p-value p(X ) = Φ(T (X )) T=0.5 p=0.31
−4
−2
0
2
4
T=2 p=0.023
−4
−2
0
2
4
Test of level α: reject H0 if p(X ) ≤ α Risk under H0 : P(p(X ) ≤ α) ≤ α Etienne Roquain
Contributions to multiple testing theory
Introduction
3 / 37
Massive and complex data I Microarray interesting genes?
I Neuroimaging (FMRI) activated regions?
Etienne Roquain
I Astronomy directions with stars?
Contributions to multiple testing theory
Introduction
4 / 37
Multiple testing Pure noise, m = 48 indep. tests:
At least one false positive: with prob. 1 − (1 − 0.05)48 ≥ 0.91 Etienne Roquain
Contributions to multiple testing theory
Introduction
5 / 37
Multiple testing Pure noise, m = 192 indep. tests:
At least one false positive: with prob. 1 − (1 − 0.05)192 ≥ 1 − 6 × 10−5 Etienne Roquain
Contributions to multiple testing theory
Introduction
5 / 37
Young’s false positive rules
(i) With enough testing, false positives will occur. (ii) Internal evidence will not contradict a false positive result. (iii) Good investigators will come up with a possible explanation. (iv) It only happens to the other persons.
Etienne Roquain
Contributions to multiple testing theory
Introduction
6 / 37
General setting I Data X ∈ (X , X) with X ∼ P ∈ P (model) I H0,i : “P ∈ P0,i ", 1 ≤ i ≤ m, null hypotheses for P I true/false label θ = θ(P) ∈ {0, 1}m such that θi = 0 if and only if H0,i is true for P I Assume p-values (pi (X ), 1 ≤ i ≤ m) such that if θi = 0, pi (X ) U(0, 1) if θi = 1, pi (X ) ∼ let arbitrary (typically "small")
Goal Recover θ from (pi (X ), 1 ≤ i ≤ m)
Etienne Roquain
Contributions to multiple testing theory
Introduction
7 / 37
1.0
Thresholding
0
0.8
0 0
0
0 0
0
0
0 0 0
0
0
0 0
0.6
0 0
0
0
0
0.4
pi
0 0
0
0
0
0 0
0
0
0
0
0 0
0
0
0
0.2
0 0 0
0.0
0
0
0
0
0
0
0
0
0 0
20
40
60
80
0
0
100
i
I Rejection set R = {1 ≤ i ≤ m : pi (X ) ≤ bt} I Some false positives Etienne Roquain
Contributions to multiple testing theory
Introduction
8 / 37
1.0
Thresholding
0
0.8
0 0
0
0 0
0
0
0 0 0
0
0
0 0
0.6
0 0
0
0
0
0.4
pi
0 0
0
0
0
0 0
0
0
0
0
0 0
0
0
0
0.2
0 0 0
0.0
0
0
0
0
0
0
0
0
0 0
20
40
60
80
0
0
100
i
I Rejection set R = {1 ≤ i ≤ m : pi (X ) ≤ bt} I Some false positives Etienne Roquain
Contributions to multiple testing theory
Introduction
8 / 37
1.0
Thresholding
0
0.8
0 0
0
0 0
0
0
0 0 0
0
0
0 0
0.6
0 0
0
0
0
0.4
pi
0 0
0
0
0
0 0
0
0
0
0
0
0.2
0 0 ●
0.0
0 ●
0 ●
0 ●
20
0
0 ●
0 ●
0 ●
0 ●
0 ●
0
0
0
0
40
60
0 ●
80
0 ●
0 ● ● 0
100
i
I Rejection set R = {1 ≤ i ≤ m : pi (X ) ≤ bt} I Some false positives Etienne Roquain
Contributions to multiple testing theory
Introduction
8 / 37
1.0
Ordering p-values
0
0.8
0 0
0
0 0
0
0
0 0 0
0
0
0 0
0.6
0 0
0
0
0
0.4
pi
0 0
0
0
0
0 0
0
0
0
0
0 0
0
0
0
0.2
0 0 0
0.0
0
0
0
0
0
0
0
0
0 0
20
40
60
80
0
0
100
i
I Stopping rule I Reject `b smallest p-values Etienne Roquain
Contributions to multiple testing theory
Introduction
9 / 37
1.0
Ordering p-values
0.8
00 0 0000
0.6
000 0
00 0 00
00 00
0.4
p(l)
0
00
000 0 0 00 00
0 0
00
0.2
0 0 0000
0.0
0
0
0 00
20
00
00
40
60
80
100
l
I Stopping rule I Reject `b smallest p-values Etienne Roquain
Contributions to multiple testing theory
Introduction
9 / 37
1.0
Ordering p-values
0.8
00 0 0000
0.6
000 0
00 0 00
00 00
0.4
p(l)
0
00
000 0 0 00 00
0 0
00
0.2
0 0 0000
0.0
0
0
0 00
20
00
00
40
60
80
100
l
I Stopping rule I Reject `b smallest p-values Etienne Roquain
Contributions to multiple testing theory
Introduction
9 / 37
1.0
Ordering p-values
0.8
00 0 0000
0.6
000 0
00 0 00
00 00
0.4
p(ell)
0
00
000 0 0 00 00
0 0
00
0.2
0
0.0
0 ●
0
0 0● 0● 0 ● 000 ●●● ●
20
0 0● ●
● 00 ●
0 ●
40
60
80
100
l
I Stopping rule I Reject `b smallest p-values Etienne Roquain
Contributions to multiple testing theory
Introduction
9 / 37
1.0
Choosing procedure?
0.6
0.8
0 00 000 0 000 00000 0 000
000
0.4
00 0
000 000
000 0 0
0.0
0.2
0 00
0
0
0
00
0
20
00 00 0
0
40
60
80
100
I "Quality" criterion? Etienne Roquain
Contributions to multiple testing theory
Introduction
10 / 37
Family-wise error rate (FWER) For a threshold bt, FWER(bt, P) = P(∃i ∈ {1, . . . , m} : θi = 0, pi ≤ bt) = P inf {pi } ≤ bt i:θi =0
FWER control α given, find bt = btα with ∀P ∈ P, FWER(bt, P) ≤ α,
I Clear interpretation I Bonferroni bt = α/m (union bound) I Refinements [Holm, 1979], [Romano and Wolf, 2005], ... Etienne Roquain
Contributions to multiple testing theory
Introduction
11 / 37
Family-wise error rate (FWER) For a threshold bt, FWER(bt, P) = P(∃i ∈ {1, . . . , m} : θi = 0, pi ≤ bt) = P inf {pi } ≤ bt i:θi =0
FWER control α given, find bt = btα with ∀P ∈ P, FWER(bt, P) ≤ α,
I Clear interpretation I Bonferroni bt = α/m (union bound) I Refinements [Holm, 1979], [Romano and Wolf, 2005], ... Etienne Roquain
Contributions to multiple testing theory
Introduction
11 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100 00 0000
0.6
0.8
0 0 0 000 0 0
0
0
00
0
0.2
0.4
0
0.0
0
0
0
0
0
20
Etienne Roquain
0
0 00
0 00 0000 0
0 00 00 0 00 00
000 0 0
0
Noncorrected
00 Bonferroni
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0.8
0 00 000 0 0 0 000 00 00 00 0
0.4
0.6
0 0
0.2
00
0 00
0
0 00 000 00
00 0 0 0 0 0 0
00 00 0 00
0.0
0 0
0
20
Etienne Roquain
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0
0.6
0.8
00
000
0.4
0
00 0000 00 0
0 0 000
00 00 00 0
00
0 00 0 00 0 00 0
0.2
0 00 0 0 0
0.0
0 0
0
000
20
Etienne Roquain
0 0
0
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
0.6
0.8
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0 000
0
0
000
0 0000 0 0
0.4
0
0 00
00 0 0000 00 000 0
000 000
0.2
0
0.0
0 00 00 0
0
0
0
20
Etienne Roquain
00
0 00
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0.6
0.8
00 000 0 0
0 00
0
0 0.4
0 000
0.2
0 000 0 00 0
0
000 000 0 000 0 0 0 00 0 000
00 0
0.0
000 0
0
0 00
20
Etienne Roquain
0
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0
0.6
0.8
000 0 0000
0
0
00
00 00 00 0 0
0.0
0.2
0.4
0 0
0 0
00
0
0
20
Etienne Roquain
0 0
0 0
0 00 00 0 0 00
0
40
0
0
0 0 00
00 0 0
60
Contributions to multiple testing theory
0000
80
100
Introduction
12 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0.8
0 0 00 0 00 00 0
0.6
00 00000 0 0000 0 000 0 0 00 0
0.4
00 000
0.2
000 0 0 0
0
00 0
0 0
0
0.0
0 00
0
20
Etienne Roquain
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
0.8
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0.6
0
0
00 0 0 00000
0 0.4
0000 0 0 0 0
0.2
00 0000 0
00 0 0 0 000
0 0
0
00
00 00
0
00 0 00
0.0
00 0
0
20
Etienne Roquain
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0.8
00 000000 0 0 0 000
0 00
0.6
0 0 00
00
0 0.4
00 00 000
0 0 0 00
00 0
0.2
0 0
0.0
0 0
0
0
0
00
20
Etienne Roquain
0 0
0
00
0
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
1.0
FWER(Bonf.) illustration, α = 0.2, m = 100
0.8
000 00 000
0.6
00000 00 0 000
0.4
0
0.2
0000 0 00
0.0
0
0
0 00
00 000
0 0
00 0000 0 00 0
0
0
0
20
Etienne Roquain
40
60
Contributions to multiple testing theory
80
100
Introduction
12 / 37
False discovery rate (FDR) What should we choose? I 4 false positives out of 10 positives? I 7 false positives out of 100 positives? For a threshold bt, FDR(bt, P) = E[FDP(bt, P)],
#{i : θi = 0, pi ≤ bt} FDP(bt, P) = #{i : pi ≤ bt}
0 =0 0
FDR control α given, find bt = btα with ∀P ∈ P, FDR(bt, P) ≤ α, If FDR ≤ 0.05: I if 20 rejections, allow at most 1 false discovery (on average) I if 1000 rejections, allow at most 50 false discoveries (on average) Etienne Roquain
Contributions to multiple testing theory
Introduction
13 / 37
False discovery rate (FDR) What should we choose? I 4 false positives out of 10 positives? I 7 false positives out of 100 positives? For a threshold bt, FDR(bt, P) = E[FDP(bt, P)],
#{i : θi = 0, pi ≤ bt} FDP(bt, P) = #{i : pi ≤ bt}
0 =0 0
FDR control α given, find bt = btα with ∀P ∈ P, FDR(bt, P) ≤ α, If FDR ≤ 0.05: I if 20 rejections, allow at most 1 false discovery (on average) I if 1000 rejections, allow at most 50 false discoveries (on average) Etienne Roquain
Contributions to multiple testing theory
Introduction
13 / 37
False discovery rate (FDR) What should we choose? I 4 false positives out of 10 positives? I 7 false positives out of 100 positives? For a threshold bt, FDR(bt, P) = E[FDP(bt, P)],
#{i : θi = 0, pi ≤ bt} FDP(bt, P) = #{i : pi ≤ bt}
0 =0 0
FDR control α given, find bt = btα with ∀P ∈ P, FDR(bt, P) ≤ α, If FDR ≤ 0.05: I if 20 rejections, allow at most 1 false discovery (on average) I if 1000 rejections, allow at most 50 false discoveries (on average) Etienne Roquain
Contributions to multiple testing theory
Introduction
13 / 37
False discovery rate (FDR) What should we choose? I 4 false positives out of 10 positives? I 7 false positives out of 100 positives? For a threshold bt, FDR(bt, P) = E[FDP(bt, P)],
#{i : θi = 0, pi ≤ bt} FDP(bt, P) = #{i : pi ≤ bt}
0 =0 0
FDR control α given, find bt = btα with ∀P ∈ P, FDR(bt, P) ≤ α, If FDR ≤ 0.05: I if 20 rejections, allow at most 1 false discovery (on average) I if 1000 rejections, allow at most 50 false discoveries (on average) Etienne Roquain
Contributions to multiple testing theory
Introduction
13 / 37
BH procedure
[Benjamini and Hochberg, 1995]
`b = max{0 ≤ ` ≤ m : p(`) ≤ α`/m}
with
1.0
bt = α(`b ∨ 1)/m
● ● ● ● ●
● ●
● ●
0.8
● ● ●
● ●
0.6
●
●
●
●
0.4
● ● ● ● ● ● ● ● ● ●
● ●
● ●
Noncorrected
0.2
● ● ● ● ● ● ●
0.0
● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
0
● ●
20
Etienne Roquain
● ● ● ● ● ● ● ● ● ● ● ● ● ●
●
● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ● ● ● ●
BH
● ● ● ●
Bonferroni
40
60
Contributions to multiple testing theory
80
100
Introduction
14 / 37
1.0
FDR(BH) illustration, α = 0.2, m = 100 FDP = 0.14
00 0000
0.8
0 0 0 000 0 0
0.2
0.4
0.6
00
0.0
0
0
0
0
0
20
Etienne Roquain
0
0 00
0 00 0000 0
0 00 00 0 00 00
000 0 0
0
00
40
60
Contributions to multiple testing theory
80
100
Introduction
15 / 37
1.0
FDR(BH) illustration, α = 0.2, m = 100 FDP = 0
0.8
0 00 000
0 00 000 00
0.4
0.6
0 0 0 000 00 00 00 0
0.2
00
0 00
0
00 0 0 0 0 0 0
00 00 0 00
0.0
0 0
0
20
Etienne Roquain
40
60
Contributions to multiple testing theory
80
100
Introduction
15 / 37
1.0
FDR(BH) illustration, α = 0.2, m = 100 FDP = 0.19
0
0.4
0.6
0.8
00
00 0000 00 0
0 0 000
00 00 00 0
00
0 00 0 00 0 00 0
0.2
0 00 0 0 0
0.0
0 0
0
000
20
Etienne Roquain
0 0
0
40
60
Contributions to multiple testing theory
80
100
Introduction
15 / 37
1.0
FDR(BH) illustration, α = 0.2, m = 100
0.6
0.8
FDP = 0.094
0 000
0 00
000
00 0 0000 00 000 0
0.4
0 0000 0 0 000 000
0.2
0
0.0
0 00 00 0
0
0
0
20
Etienne Roquain
00
0 00
40
60
Contributions to multiple testing theory
80
100
Introduction
15 / 37
1.0
FDR(BH) illustration, α = 0.2, m = 100 FDP = 0.18
0.6
0.8
00 000 0 0
0.4
0 000
0.2
0 000 0 00 0
0
000 000 0 000 0 0 0 00 0 000
00 0
0.0
000 0
0
0 00
20
Etienne Roquain
0
40
60
Contributions to multiple testing theory
80
100
Introduction
15 / 37
FDR control for BH Theorem [Benjamini and Hochberg (1995)] [Benjamini and Yekutieli (2001)] Assume I pi ∼ U(0, 1) for θi = 0 I (pi , 1 ≤ i ≤ m) are mutually indep for any P ∈ P. Then for all P ∈ P, FDR(BHα , P) = π0 α ≤ α for π0 proportion of zeros in θ(P). Many extensions (weighting, step-up-down, PRDS) with devoted proofs Blanchard and R. (2008). Two simple sufficient conditions for FDR control. EJS. Blanchard, Delattre and R. (2014). Testing over a continuum of null hypotheses[...]. Bernoulli. Picard, R., Fougeres, Reynaud-Bouret. Comparing Poisson Proc. intensities by continuous testing. Soon.
Etienne Roquain
Contributions to multiple testing theory
Introduction
16 / 37
FDR control for BH Theorem [Benjamini and Hochberg (1995)] [Benjamini and Yekutieli (2001)] Assume I pi ∼ U(0, 1) for θi = 0 I (pi , 1 ≤ i ≤ m) are mutually indep for any P ∈ P. Then for all P ∈ P, FDR(BHα , P) = π0 α ≤ α for π0 proportion of zeros in θ(P). Many extensions (weighting, step-up-down, PRDS) with devoted proofs Blanchard and R. (2008). Two simple sufficient conditions for FDR control. EJS. Blanchard, Delattre and R. (2014). Testing over a continuum of null hypotheses[...]. Bernoulli. Picard, R., Fougeres, Reynaud-Bouret. Comparing Poisson Proc. intensities by continuous testing. Soon.
Etienne Roquain
Contributions to multiple testing theory
Introduction
16 / 37
1 Introduction
Motivation Setting Classical results
2 Adaptive procedures
Adaptive control Adaptive error rate Adaptive p-values Adaptive classification
3 Outlook
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
17 / 37
1 Introduction
Motivation Setting Classical results
2 Adaptive procedures
Adaptive control Adaptive error rate Adaptive p-values Adaptive classification
3 Outlook
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
18 / 37
Adaptation to π0 FDR(BHα ) = π0 α ≤ α If π0 is known: for ϑ = (π0 )
gap if π0 not close to 1
−1
FDR(BHϑα ) = π0 (ϑα) = α but ϑ = (π0 )−1 often unknown. b If π0 is unknown: data-driven procedure BHϑα b for ϑ “estimating" ϑ such that I FDR(BHϑα b )≤α I BHϑα b making more discoveries Many work [Schweder and Spjøtvoll (1982)], [...], [Benjamini et al. (2006)] Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR.
(1 − α)m ϑb = Pm i=1 1{pi > α} + 1 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
19 / 37
Adaptation to π0 FDR(BHα ) = π0 α ≤ α
gap if π0 not close to 1
−1
If π0 is known: for ϑ = (π0 )
FDR(BHϑα ) = π0 (ϑα) = α but ϑ = (π0 )−1 often unknown. b If π0 is unknown: data-driven procedure BHϑα b for ϑ “estimating" ϑ such that I FDR(BHϑα b )≤α I BHϑα b making more discoveries Many work [Schweder and Spjøtvoll (1982)], [...], [Benjamini et al. (2006)] Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR.
(1 − α)m ϑb = Pm i=1 1{pi > α} + 1 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
19 / 37
Adaptation to π0 FDR(BHα ) = π0 α ≤ α
gap if π0 not close to 1
−1
If π0 is known: for ϑ = (π0 )
FDR(BHϑα ) = π0 (ϑα) = α but ϑ = (π0 )−1 often unknown. b If π0 is unknown: data-driven procedure BHϑα b for ϑ “estimating" ϑ such that I FDR(BHϑα b )≤α I BHϑα b making more discoveries Many work [Schweder and Spjøtvoll (1982)], [...], [Benjamini et al. (2006)] Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR.
(1 − α)m ϑb = Pm i=1 1{pi > α} + 1 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
19 / 37
Adaptation to π0 FDR(BHα ) = π0 α ≤ α
gap if π0 not close to 1
−1
If π0 is known: for ϑ = (π0 )
FDR(BHϑα ) = π0 (ϑα) = α but ϑ = (π0 )−1 often unknown. b If π0 is unknown: data-driven procedure BHϑα b for ϑ “estimating" ϑ such that I FDR(BHϑα b )≤α I BHϑα b making more discoveries Many work [Schweder and Spjøtvoll (1982)], [...], [Benjamini et al. (2006)] Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR.
(1 − α)m ϑb = Pm i=1 1{pi > α} + 1 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
19 / 37
Some contributions for adaptive control Gaussian model: X ∼ P = N (µ, Γ), testing µi = 0 (≤ 0) against µi 6= 0 (> 0) I Adapt. to unknown ϑ = π0−1 for FDR and Γ = Im or equi Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR. ⇒ Control
I Adapt. to unknown ϑ = Γ for FWER Arlot et al. (2010a), (2010b). AoS. ⇒ Control; gap
I Adapt. to known ϑ = Γ for quantile of FDP Delattre and R. (2015). New procedures controlling FDP [...]. AoS. ⇒ Control; gap
I Adapt. to known ϑ = µ for FDR and Γ = Im R. and van de Wiel (2009). Optimal weighting for FDR control. EJS. ⇒ Control and optimality Perf(Rϑ , P) ' maxR∈WBH {Perf(R, P)}
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
20 / 37
Some contributions for adaptive control Gaussian model: X ∼ P = N (µ, Γ), testing µi = 0 (≤ 0) against µi 6= 0 (> 0) I Adapt. to unknown ϑ = π0−1 for FDR and Γ = Im or equi Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR. ⇒ Control
I Adapt. to unknown ϑ = Γ for FWER Arlot et al. (2010a), (2010b). AoS. ⇒ Control; gap
I Adapt. to known ϑ = Γ for quantile of FDP Delattre and R. (2015). New procedures controlling FDP [...]. AoS. ⇒ Control; gap
I Adapt. to known ϑ = µ for FDR and Γ = Im R. and van de Wiel (2009). Optimal weighting for FDR control. EJS. ⇒ Control and optimality Perf(Rϑ , P) ' maxR∈WBH {Perf(R, P)}
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
20 / 37
Some contributions for adaptive control Gaussian model: X ∼ P = N (µ, Γ), testing µi = 0 (≤ 0) against µi 6= 0 (> 0) I Adapt. to unknown ϑ = π0−1 for FDR and Γ = Im or equi Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR. ⇒ Control
I Adapt. to unknown ϑ = Γ for FWER Arlot et al. (2010a), (2010b). AoS. ⇒ Control; gap
I Adapt. to known ϑ = Γ for quantile of FDP Delattre and R. (2015). New procedures controlling FDP [...]. AoS. ⇒ Control; gap
I Adapt. to known ϑ = µ for FDR and Γ = Im R. and van de Wiel (2009). Optimal weighting for FDR control. EJS. ⇒ Control and optimality Perf(Rϑ , P) ' maxR∈WBH {Perf(R, P)}
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
20 / 37
Some contributions for adaptive control Gaussian model: X ∼ P = N (µ, Γ), testing µi = 0 (≤ 0) against µi 6= 0 (> 0) I Adapt. to unknown ϑ = π0−1 for FDR and Γ = Im or equi Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR. ⇒ Control
I Adapt. to unknown ϑ = Γ for FWER Arlot et al. (2010a), (2010b). AoS. ⇒ Control; gap
I Adapt. to known ϑ = Γ for quantile of FDP Delattre and R. (2015). New procedures controlling FDP [...]. AoS. ⇒ Control; gap
I Adapt. to known ϑ = µ for FDR and Γ = Im R. and van de Wiel (2009). Optimal weighting for FDR control. EJS. ⇒ Control and optimality Perf(Rϑ , P) ' maxR∈WBH {Perf(R, P)}
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
20 / 37
Some contributions for adaptive control Gaussian model: X ∼ P = N (µ, Γ), testing µi = 0 (≤ 0) against µi 6= 0 (> 0) I Adapt. to unknown ϑ = π0−1 for FDR and Γ = Im or equi Blanchard and R. (2008). Adaptive FDR control under indep. and dep. JMLR. ⇒ Control
I Adapt. to unknown ϑ = Γ for FWER Arlot et al. (2010a), (2010b). AoS. ⇒ Control; gap
I Adapt. to known ϑ = Γ for quantile of FDP Delattre and R. (2015). New procedures controlling FDP [...]. AoS. ⇒ Control; gap
I Adapt. to known ϑ = µ for FDR and Γ = Im R. and van de Wiel (2009). Optimal weighting for FDR control. EJS. ⇒ Control and optimality Perf(Rϑ , P) ' maxR∈WBH {Perf(R, P)}
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
20 / 37
1 Introduction
Motivation Setting Classical results
2 Adaptive procedures
Adaptive control Adaptive error rate Adaptive p-values Adaptive classification
3 Outlook
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
21 / 37
Copy number alterations (CNA) in cancer cells
Normal cells
Tumour cells
CNA is useful to study various type of cancers
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
22 / 37
Data: [Chin et al. (2006)] I Preprocessing: 0 normal ; −1 deletion ; 1 amplification Tissue 1 (ER-)
Tissue 2 (ER+)
region1 region2 region3 region4 region5 region6 region7 region8 region9
0 1 -1 0 0 0 0 0 1 -1 0 0 0 0
0 -1 0 0 -1 -1 0 0 -1 0 0 1 -1 0
1 0 1 0 0 0 0 1 1 1 0 0 0 -1
1 0 0 0 0 1 1 1 0 0 0 0 1 1
1 1 1 1 1 1 1 1 1 1 0 -1 1 1
1 0 1 0 0 0 0 1 0 1 1 0 0 0
-1 1 -1 -1 1 1 1 -1 1 -1 -1 1 1 1
0 -1 0 0 -1 -1 -1 0 -1 0 0 -1 -1 -1
0 0 1 0 0 0 0 0 0 1 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 -1 -1 -1 0 0 0 0 -1 -1 -1 0 1
0 0 0 0 0 0 0 0 0 0 0 0 -1 1
0 1 0 0 0 0 0 0 1 0 0 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1 -1 1
-1 -1 0 0 0 -1 -1 -1 -1 0 0 0 -1 -1
0 0 1 1 1 0 0 0 0 1 1 1 0 0
0 -1 0 0 0 0 0 0 -1 0 0 0 0 0
0 0 -1 -1 -1 0 0 0 0 -1 -1 -1 0 0
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
regionm
-1 -1
1 0
1 0
1 1
1 1
1 1
-1 -1
-1 -1
0 0
0 0
0 0
0 0
0 0
1 1
0 -1
0 0
0 0
0 0
I Finding regions that are different between samples?
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
23 / 37
Data: [Chin et al. (2006)] I Correlations:
I Entities to be tested? Clusters or regions ? Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
23 / 37
New approach I Clustering algorithm region1 region2
0 1
0 -1
1 0
1 0
1 1
1 0
-1 1
0 -1
0 0
0 0
0 0
0 0
0 1
1 1
-1 -1
0 0
0 -1
0 0
region3 region4 region5
-1 0 0
0 0 -1
1 0 0
0 0 0
1 1 1
1 0 0
-1 -1 1
0 0 -1
1 0 0
0 0 0
-1 -1 -1
0 0 0
0 0 0
1 1 1
0 0 0
1 1 1
0 0 0
-1 -1 -1
region6
0
-1
0
1
1
0
1
-1
0
0
0
0
0
1
-1
0
0
0
region7 region8 region9
0 0 1 -1 0
0 0 -1 0 0
0 1 1 1 0
1 1 0 0 0
1 1 1 1 0
0 1 0 1 1
1 -1 1 -1 -1
-1 0 -1 0 0
0 0 0 1 0
0 0 0 0 0
0 0 0 -1 -1
0 0 0 0 0
0 0 1 0 0
1 1 1 1 1
-1 -1 -1 0 0
0 0 0 1 1
0 0 -1 0 0
0 0 0 -1 -1
0 0 0
1 -1 0
0 0 -1
0 1 1
-1 1 1
0 0 0
1 1 1
-1 -1 -1
0 0 0
0 0 0
-1 0 1
0 -1 1
0 0 0
1 -1 1
0 -1 -1
1 0 0
0 0 0
-1 0 0
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
regionm
-1 -1
1 0
1 0
1 1
1 1
1 1
-1 -1
-1 -1
0 0
0 0
0 0
0 0
0 0
1 1
0 -1
0 0
0 0
0 -1
I FWER for regions ∪ clusters Kim, R., van de Wiel (2010). Spatial clustering of array CGH features [...]. SAGMB.
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
24 / 37
1 Introduction
Motivation Setting Classical results
2 Adaptive procedures
Adaptive control Adaptive error rate Adaptive p-values Adaptive classification
3 Outlook
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
25 / 37
Data: [Hedenfalk et al. (2001)]
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
26 / 37
Data: [Hedenfalk et al. (2001)] Factor model
Etienne Roquain
Xi = µi +
P
h
Wh ci,h + ξi
[Friguet et al. (2009)]
Contributions to multiple testing theory
Adaptive procedures
26 / 37
Data: [Hedenfalk et al. (2001)] Known equi-correlation
Etienne Roquain
Xi = µi + W + ξi
Contributions to multiple testing theory
Adaptive procedures
26 / 37
p-values under Gaussian equi-correlation
0 00 0 00000
000000000000000000000
000 0000000 000 000 00000
W = −1
0.8
0.8
1.0
000
0000 0 0 000 0 00
000 00
00
00 00 000
0.6
00
0.4
0
0.2
0.8
000 0 000 00
W = −2
0.6
0
0.4
W= 0
0.2
1.0
1.0
Xi = µi + W + ξi ,
0 0 00
0 000 0000
00
1.0
0.4
20
40
60
80
100
0
1.0
0
000
00 0
0.0
0.0
0.6
00 0 00 0000 00
W= 1
20
40
60
80
100
W= 2
20
60
80
100
0.4
0
0000 0 00 00 0 000 0000
0
20
40
00 0000000
60
0.2
0.2
40
0.6
0.8 0.6 0.4
00 00 00 00 0 00000 000 000
0.0
0
0
0 00
0.0
0.2 0.0
00 0 00
0000 00 0 00 0 00
0.8
00 00 00 000
80
100
0 00 00 0 000 00000000 0000000 00 000
0
20
40
60
0 00 0000 00 000 00000000
80
100
c = X(1) − E(ξ(1) ) I W c is ≤ C/(log m) I Max risk W I Minimax risk ≥ c/(log m)5 Etienne Roquain
Contributions to multiple testing theory
Delattre and R. Soon. Adaptive procedures
27 / 37
p-values under Gaussian equi-correlation
100
60
0.0
60
80
100
80
100
0 000 0000 00 000 00 0000 00
W= 2
000 00 00 00 000 0 000 000 00 0 00
0
40
40
0.4
00
What = 1.6 20
20
0.2
0 000 000 00 0 00
0
What = −0.41 0
0.6
000 00
00
00
00
00
What = 2.6
0.0
80
100
0 000 0000 00 000 00 0000 00
0.0
60
0.6 80
00 00 000
0.2
0.0
40
0.4
60
W= 1
0
20
40
1.0
20
0.6
00
0 0 00 00
0 000 0000
0.8
0.0 1.0
0 000 0000
000 00 00 00 000
0
What = −1.4 0
What = 0.59 0
00
0.4
0.2
00
0 000 0000
0.8
0.4
00 00 000
0
00
000 00 0000 00
0.2
0 0 00
0
00
0 000 0000 00
W = −1
0.8
0.8
000 00 00 00 000
0.2
000
0.6
000 00 0000 00
0.4
0.8
000 00 0000 00
0 0 00
0 000 0000 00
W = −2
0.6
1.0
0 000 0000 00
W= 0
1.0
c = µi + W − W c + ξi , Xi − W 1.0
Xi = µi + W + ξi ,
0
20
40
60
80
100
c = X(1) − E(ξ(1) ) I W c is ≤ C/(log m) I Max risk W I Minimax risk ≥ c/(log m)5 Etienne Roquain
Contributions to multiple testing theory
Delattre and R. Soon. Adaptive procedures
27 / 37
p-values under Gaussian equi-correlation
100
60
0.0
60
80
100
80
100
0 000 0000 00 000 00 0000 00
W= 2
000 00 00 00 000 0 000 000 00 0 00
0
40
40
0.4
00
What = 1.6 20
20
0.2
0 000 000 00 0 00
0
What = −0.41 0
0.6
000 00
00
00
00
00
What = 2.6
0.0
80
100
0 000 0000 00 000 00 0000 00
0.0
60
0.6 80
00 00 000
0.2
0.0
40
0.4
60
W= 1
0
20
40
1.0
20
0.6
00
0 0 00 00
0 000 0000
0.8
0.0 1.0
0 000 0000
000 00 00 00 000
0
What = −1.4 0
What = 0.59 0
00
0.4
0.2
00
0 000 0000
0.8
0.4
00 00 000
0
00
000 00 0000 00
0.2
0 0 00
0
00
0 000 0000 00
W = −1
0.8
0.8
000 00 00 00 000
0.2
000
0.6
000 00 0000 00
0.4
0.8
000 00 0000 00
0 0 00
0 000 0000 00
W = −2
0.6
1.0
0 000 0000 00
W= 0
1.0
c = µi + W − W c + ξi , Xi − W 1.0
Xi = µi + W + ξi ,
0
20
40
60
80
100
c = X(1) − E(ξ(1) ) I W c is ≤ C/(log m) I Max risk W I Minimax risk ≥ c/(log m)5 Etienne Roquain
Contributions to multiple testing theory
Delattre and R. Soon. Adaptive procedures
27 / 37
p-values under Gaussian equi-correlation
100
60
0.0
60
80
100
80
100
0 000 0000 00 000 00 0000 00
W= 2
000 00 00 00 000 0 000 000 00 0 00
0
40
40
0.4
00
What = 1.6 20
20
0.2
0 000 000 00 0 00
0
What = −0.41 0
0.6
000 00
00
00
00
00
What = 2.6
0.0
80
100
0 000 0000 00 000 00 0000 00
0.0
60
0.6 80
00 00 000
0.2
0.0
40
0.4
60
W= 1
0
20
40
1.0
20
0.6
00
0 0 00 00
0 000 0000
0.8
0.0 1.0
0 000 0000
000 00 00 00 000
0
What = −1.4 0
What = 0.59 0
00
0.4
0.2
00
0 000 0000
0.8
0.4
00 00 000
0
00
000 00 0000 00
0.2
0 0 00
0
00
0 000 0000 00
W = −1
0.8
0.8
000 00 00 00 000
0.2
000
0.6
000 00 0000 00
0.4
0.8
000 00 0000 00
0 0 00
0 000 0000 00
W = −2
0.6
1.0
0 000 0000 00
W= 0
1.0
c = µi + W − W c + ξi , Xi − W 1.0
Xi = µi + W + ξi ,
0
20
40
60
80
100
c = X(1) − E(ξ(1) ) I W c is ≤ C/(log m) I Max risk W I Minimax risk ≥ c/(log m)5 Etienne Roquain
Contributions to multiple testing theory
Delattre and R. Soon. Adaptive procedures
27 / 37
1 Introduction
Motivation Setting Classical results
2 Adaptive procedures
Adaptive control Adaptive error rate Adaptive p-values Adaptive classification
3 Outlook
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
28 / 37
BHα adapts to the signal X ∼ N (µ, Im ) Strong signal strength
X
µ
Low signal strength
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
29 / 37
BHα adapts to the signal Low signal strength
Bonf. Etienne Roquain
BH
Uncorr.
Strong signal strength
Bonf.
Contributions to multiple testing theory
BH
Uncorr. Adaptive procedures
29 / 37
BHα useful outside multiple testing? I X ∼ N (µ, Im ) ˆ θ) I Inference on θ = ψ(µ), for some risk R(θ,
1
Estimation: θ = µ,
θbBH = Xi 1{pi ≤ bt}
[Abramovich et al. (2006)], [Donoho and Jin (2006)] 2
Classification. θi = 1{µi 6= 0}, θˆiBH = 1{pi ≤ bt} [Bogdan et al. (2011)] Neuvial and R. (2012). On FDR thresholding for classification under sparsity. AoS.
ˆ θ) as m → ∞ R(θˆBH , θ) ' infθˆ R(θ, for all θ ∈ Θsparse,m ensuring π0 ∼ 1 − m−β
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
30 / 37
BHα useful outside multiple testing? I X ∼ N (µ, Im ) ˆ θ) I Inference on θ = ψ(µ), for some risk R(θ,
1
Estimation: θ = µ,
θbBH = Xi 1{pi ≤ bt}
[Abramovich et al. (2006)], [Donoho and Jin (2006)] 2
Classification. θi = 1{µi 6= 0}, θˆiBH = 1{pi ≤ bt} [Bogdan et al. (2011)] Neuvial and R. (2012). On FDR thresholding for classification under sparsity. AoS.
ˆ θ) as m → ∞ R(θˆBH , θ) ' infθˆ R(θ, for all θ ∈ Θsparse,m ensuring π0 ∼ 1 − m−β
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
30 / 37
BHα useful outside multiple testing? I X ∼ N (µ, Im ) ˆ θ) I Inference on θ = ψ(µ), for some risk R(θ,
1
Estimation: θ = µ,
θbBH = Xi 1{pi ≤ bt}
[Abramovich et al. (2006)], [Donoho and Jin (2006)] 2
Classification. θi = 1{µi 6= 0}, θˆiBH = 1{pi ≤ bt} [Bogdan et al. (2011)] Neuvial and R. (2012). On FDR thresholding for classification under sparsity. AoS.
ˆ θ) as m → ∞ R(θˆBH , θ) ' infθˆ R(θ, for all θ ∈ Θsparse,m ensuring π0 ∼ 1 − m−β
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
30 / 37
BHα useful outside multiple testing? I X ∼ N (µ, Im ) ˆ θ) I Inference on θ = ψ(µ), for some risk R(θ,
1
Estimation: θ = µ,
θbBH = Xi 1{pi ≤ bt}
[Abramovich et al. (2006)], [Donoho and Jin (2006)] 2
Classification. θi = 1{µi 6= 0}, θˆiBH = 1{pi ≤ bt} [Bogdan et al. (2011)] Neuvial and R. (2012). On FDR thresholding for classification under sparsity. AoS.
ˆ θ) as m → ∞ R(θˆBH , θ) ' infθˆ R(θ, for all θ ∈ Θsparse,m ensuring π0 ∼ 1 − m−β
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
30 / 37
BHα as a classifier: setting Model: (Xi , θi ) ∈ R × {0, 1}, m i.i.d. random pairs (i) we observe X1 , . . . , Xm and not the “labels” θ1 , . . . , θm ; (ii) D(X1 | θ1 = 0) = N (0, 1), D(X1 | θ1 = 1) = N (∆, 1) Risk for a classifier θb ∈ {0, 1}m , m X b θ) = E m−1 R(θ, 1{θbi 6= θi } /(1 − π0 ) i=1
Procedures: I Bayes rule θb? (β, ∆) that minimizes R I BH rules θˆBH decides 1 if rejected 0 otherwise Assumptions: I (Sparsity) π0 = P(θ1 = 0) ∼ 1 − m−β , β ∈ (0, 1) I (Bayes pow) power of θb? (β, ∆) non trivial ⇒ ∆ ∼ (2β log m)1/2 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
31 / 37
BHα as a classifier: setting Model: (Xi , θi ) ∈ R × {0, 1}, m i.i.d. random pairs (i) we observe X1 , . . . , Xm and not the “labels” θ1 , . . . , θm ; (ii) D(X1 | θ1 = 0) = N (0, 1), D(X1 | θ1 = 1) = N (∆, 1) Risk for a classifier θb ∈ {0, 1}m , m X b θ) = E m−1 R(θ, 1{θbi 6= θi } /(1 − π0 ) i=1
Procedures: I Bayes rule θb? (β, ∆) that minimizes R I BH rules θˆBH decides 1 if rejected 0 otherwise Assumptions: I (Sparsity) π0 = P(θ1 = 0) ∼ 1 − m−β , β ∈ (0, 1) I (Bayes pow) power of θb? (β, ∆) non trivial ⇒ ∆ ∼ (2β log m)1/2 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
31 / 37
BHα as a classifier: setting Model: (Xi , θi ) ∈ R × {0, 1}, m i.i.d. random pairs (i) we observe X1 , . . . , Xm and not the “labels” θ1 , . . . , θm ; (ii) D(X1 | θ1 = 0) = N (0, 1), D(X1 | θ1 = 1) = N (∆, 1) Risk for a classifier θb ∈ {0, 1}m , m X b θ) = E m−1 R(θ, 1{θbi 6= θi } /(1 − π0 ) i=1
Procedures: I Bayes rule θb? (β, ∆) that minimizes R I BH rules θˆBH decides 1 if rejected 0 otherwise Assumptions: I (Sparsity) π0 = P(θ1 = 0) ∼ 1 − m−β , β ∈ (0, 1) I (Bayes pow) power of θb? (β, ∆) non trivial ⇒ ∆ ∼ (2β log m)1/2 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
31 / 37
BHα as a classifier: setting Model: (Xi , θi ) ∈ R × {0, 1}, m i.i.d. random pairs (i) we observe X1 , . . . , Xm and not the “labels” θ1 , . . . , θm ; (ii) D(X1 | θ1 = 0) = N (0, 1), D(X1 | θ1 = 1) = N (∆, 1) Risk for a classifier θb ∈ {0, 1}m , m X b θ) = E m−1 R(θ, 1{θbi 6= θi } /(1 − π0 ) i=1
Procedures: I Bayes rule θb? (β, ∆) that minimizes R I BH rules θˆBH decides 1 if rejected 0 otherwise Assumptions: I (Sparsity) π0 = P(θ1 = 0) ∼ 1 − m−β , β ∈ (0, 1) I (Bayes pow) power of θb? (β, ∆) non trivial ⇒ ∆ ∼ (2β log m)1/2 Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
31 / 37
BHα as a classifier: result Theorem (non-asymptotic)
[Neuvial and R. (2012)]
d(0) −1 opt For αm ∈ (0, 1/2), for any m ≥ 2 such that (2 log τm )1/2 ≥ (log(αm /qm ) − log(νπ0,m (1 − ε))), we have Cm (1−ν) 0 BH ? R(θˆm , θ) − R(θˆm , θ) ≤ @
+
αm
+ d(0)
1 − αm αm /(mπ1,m ) (1 − αm )2
1 −1 opt (log(αm /qm ) − log(νπ0,m (1 − ε)))+ A (2 log τm )1/2
+e
−mπ1,m νε2 Cm /4
,
opt opt where qm = 1/αm − 1 > 1, τm = π0,m /π1,m > 1, ε, ν ∈ (0, 1).
Corollary (asymptotic)
[Neuvial and R. (2012)]
For θ satisfying (Sparsity) (Bayes pow), choosing αm ∝ 1/(log m)1/2 ensures 1 BH ? ˆ ˆ R(θ , θ) = R(θ , θ) 1 + O . (log m)1/2
Etienne Roquain
Contributions to multiple testing theory
Adaptive procedures
32 / 37
BHα as a classifier: illustration m = 1282 , π0 = 1 − m−β , ∆ = (log m)1/2 , α = 0.2 MISCEr= 0.358
MISCEr= 0.354
MISCEr= 3.53
MISCEr= 0.871
MISCEr= 0.531
MISCEr= 0.543
MISCEr= 9.24
MISCEr= 0.876
MISCEr= 0.636
MISCEr= 0.636
MISCEr= 24.3
MISCEr= 0.8
MISCEr= 0.711
MISCEr= 0.689
MISCEr= 74.1
β = 0.6
β = 0.5
β = 0.4
β = 0.3
MISCEr= 0.869
Bonf. Etienne Roquain
BH
Bayes
Contributions to multiple testing theory
Uncorr. Adaptive procedures
33 / 37
1 Introduction
Motivation Setting Classical results
2 Adaptive procedures
Adaptive control Adaptive error rate Adaptive p-values Adaptive classification
3 Outlook
Etienne Roquain
Contributions to multiple testing theory
Outlook
34 / 37
FDR revolution Benjamini and Hochberg (1995). Controlling the false discovery rate: a practical and powerful approach to
2000
2500
multiple testing
1500
GENETICS HEREDITY
1000
BIOTECHNOLOGY APPLIED MICROBIOLOGY
NEUROSCIENCES NEUROLOGY
BIOCHEMISTRY MOLECULAR BIOLOGY
OTHERS
500
COMPUTER SCIENCE MATHEMATICS
ENVIRONMENTAL SCIENCES ECOLOGY ONCOLOGY MATHEMATICAL COMPUTATIONAL BIOLOGY
0
SCIENCE TECHNOLOGY OTHER TOPICS
1996
1999
2002
2005
2008
2011
Figure: 13,427 papers citing this work (1996-2013) according to "the web of science".
Etienne Roquain
Contributions to multiple testing theory
Outlook
35 / 37
Future research Co-supervising: I Marine Roux: FDR control for two-sided test statistics I Guillermo Durand: data driven weighting Other ’hot’ topics: I Correcting p-values in factor model I False positive number in a user-designed set I Bayesian multiple testing procedures
Etienne Roquain
Contributions to multiple testing theory
Outlook
36 / 37
Thank you!
Etienne Roquain
Contributions to multiple testing theory
Outlook
37 / 37
