False Discovery Rate Part I : introduction et enjeux E. Roquain1 1

Laboratory LPMA, Université Pierre et Marie Curie (Paris 6), France

Point de Vue, 3rd February 2014

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

1 / 33

1 Introduction

2 False discovery rate control

3 FDR in other statistical issues

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

2 / 33

1 Introduction 2 False discovery rate control 3 FDR in other statistical issues

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

3 / 33

A “multiple testing joke" (http://xkcd.com)

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

4 / 33

A “multiple testing joke" (http://xkcd.com)

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

4 / 33

A “multiple testing joke" (http://xkcd.com)

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

4 / 33

A “multiple testing joke" (http://xkcd.com)

Multiplicity problem P( make at least one false discovery )  P( the i-th is a false discovery ) A correction is needed to assess significancy! E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

4 / 33

Some other examples Paradoxes due to large scale experiments Probable facts appear significant

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

5 / 33

Science-wise multiplicity issue - [Ioannidis (2005, PLoS Medicine)]

Open access, freely available online

Essay

Why Most Published Research Findings Are False John P. A. Ioannidis

Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

E. Roquain

factors that influence this problem and some corollaries thereof.

Modeling the Framework for False Positive Findings Several methodologists have pointed out [9–11] that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05. Research is not most appropriately represented and summarized by p-values, but, unfortunately, there is a widespread notion that medical research articles

It can be proven that most claimed research findings are false.

should be interpreted based only on p-values. Research findings are defined here as any relationship reaching formal statistical significance, e.g., effective interventions, informative predictors, risk factors, or associations. “Negative” research is also very useful. “Negative” is actually a misnomer, and the misinterpretation is widespread. ublished research findings are However, here we will target sometimes refuted by FDR subsequent : introduction, enjeux et perspectives. relationships that investigators claim

is characteristic of the field and can vary a lot depending on whether the field targets highly likely relationships or searches for only one or a few true relationships among thousands and millions of hypotheses that may be postulated. Let us also consider, for computational simplicity, circumscribed fields where either there is only one true relationship (among many that can be hypothesized) or the power is similar to find any of the several existing true relationships. The pre-study probability of a relationship being true is R⁄(R + 1). The probability of a study finding a true relationship reflects the power 1 − β (one minus the Type II error rate). The probability of claiming a relationship when none truly exists reflects the Type I error rate, α. Assuming that c relationships are being probed in the field, the expected values of the 2 × 2 table are given in Table 1. After a research finding has been claimed based on achieving formal statistical significance, the post-study probability that it is true is the positive predictive value, PPV. The PPV is also the complementary probability of what Wacholder et al. have called the false positive report probability [10]. According to the 2 × 2 table, one gets PPV = (1 − β)R⁄(R − βR + α). A research finding is thus

Part I.

Introduction

6 / 33

Science-wise multiplicity issue - [Ioannidis (2005, PLoS Medicine)] [Talk Benjamini Southampton (2013)] ; modeling:

Try many experiments ⇓ 1000 pure noise

30 perfect signal ⇓

publish results with a p-value ≤ 0.05 ⇓ ' 50 false discoveries I

Jager and Leek (2013). An estimate of the science-wise false discovery rate and application to the top medical literature

I

30 true discoveries

' 14%

Ioannidis (2014). Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

6 / 33

Science-wise multiplicity issue - [Ioannidis (2005, PLoS Medicine)] [Talk Benjamini Southampton (2013)] ; modeling:

Try many experiments ⇓ 1000 pure noise

30 perfect signal ⇓

publish results with a p-value ≤ 0.05 ⇓ ' 50 false discoveries I

Jager and Leek (2013). An estimate of the science-wise false discovery rate and application to the top medical literature

I

30 true discoveries

' 14%

Ioannidis (2014). Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

6 / 33

Science-wise multiplicity issue - [Ioannidis (2005, PLoS Medicine)] [Talk Benjamini Southampton (2013)] ; modeling:

Try many experiments ⇓ 1000 pure noise

30 perfect signal ⇓

publish results with a p-value ≤ 0.05 ⇓ ' 50 false discoveries I

Jager and Leek (2013). An estimate of the science-wise false discovery rate and application to the top medical literature

I

30 true discoveries

' 14%

Ioannidis (2014). Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

6 / 33

Science-wise multiplicity issue - [Ioannidis (2005, PLoS Medicine)] [Talk Benjamini Southampton (2013)] ; modeling:

Try many experiments ⇓ 1000 pure noise

30 perfect signal ⇓

publish results with a p-value ≤ 0.05 ⇓ ' 50 false discoveries I

Jager and Leek (2013). An estimate of the science-wise false discovery rate and application to the top medical literature

I

30 true discoveries

' 14%

Ioannidis (2014). Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

6 / 33

Multiplicity in microarray [Hedenfalk et al. (2001)]

BRCA1 vs BRCA2

genes

I expression level (activity) I genes differentially activated? I 1 test for each gene I thousands of genes I nb replications  dimension I correlations

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

7 / 33

Other applications I Neuroimaging (FMRI) activated regions? I Econometrics winning strategies? I Astronomy directions with stars?

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

8 / 33

Canonical setting I Xi = avg group 2 - avg group 1 (rescaled) for genes i I Gaussian model :     

X1 X2 .. .





     = µ  

Xm

H1 H2 .. .





    +  

Hm

ε1 ε2 .. .

   , 

εm

with µ > 0, H ∈ {0, 1}m (fixed) and ε ∼ N (0, Γ) (Γi,i = 1). I Γ = dependence structure = Im for now

Question: for each i, Hi = 0 or Hi = 1 ? Multiple testing : favors the “0" decision

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

9 / 33

Individual decision and errors I Test statistic: Xi I p-value: pi = Φ(Xi ), with Φ(z) = P(Z ≥ z), Z ∼ N (0, 1)

pi such that if Hi = 0, pi ∼ U(0, 1) if Hi = 1, pi ∼ Φ(Φ

−1

(·) − µ)

b i = 1{pi ≤ t} for some threshold t I Choose H I Two errors:

Hi = 0 Hi = 1

bi = 0 H true negative false negative

bi = 1 H false positive true positive

I False positive more annoying E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

10 / 33

1.0

Picture 1. m = 100; m0 = 50; µ = 2; Γ = Im

0.8

0 0 0 0 0 00

0.0

0.2

0.4

0.6

000 00 0 00 00 000 0 1 0 1 00 000 1 0 01 0 00 0 001 1 0001 0 0000 011 1 0 0 1 1 1 111111 11011 1000 111111 11111111111111111111

0.0 E. Roquain

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

11 / 33

1.0

Picture 2. m = 100; m0 = 95; µ = 3; Γ = Im

0.0

0.2

0.4

0.6

0.8

00 0 0 000 00 0 0 0 000 000 0 0 0 00 0 00 00 000 000 00 0000 00 000 0 0 0 0000 000 00 00 000 00 000 0000 0 000 000 00 00 00 0 0 0 0 0 001 1000 1110

0.0 E. Roquain

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

12 / 33

1.0

Picture 3. m = 100; m0 = 50; µ = 0.01; Γ = Im

0.0

0.2

0.4

0.6

0.8

10 100 0 1 0001 1 10 10001 1 0 111 110 1 1 0 01 1 0000 0110 1 1 0 0 10 00 0 0 0 11 0 011 1 01 00011 1 1 01 1 101 111 11 11 000 101 100 00 001 1 0 010

0.0 E. Roquain

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

13 / 33

1.0

Picture 4. m = 100; m0 = 95; µ = 0.01; Γ = Im

0.0

0.2

0.4

0.6

0.8

0 00 00 000 001 0000 00 0 000 0 0000 0 0 0000 000 0 0 0 0 0 0000 00 0000 0 0 00 00 00 0 0 00 0 0 00010 0 0 00 0 0 001 0 0100 0 00 000 0000 00 1 0000

0.0 E. Roquain

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

14 / 33

0.0

0.2

0.4

0.6

0.8

1.0

Doing like for 1 test? t ≡ α = 0.1

0.0 E. Roquain

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

15 / 33

1.0

Doing like for 1 test? t ≡ α = 0.1

0.8

0 100 0000 0 0 0 00 00 0 0 0 0 000 0000

0.0

0.2

0.4

0.6

000 0 00 0 00 00 0 00 000 00000 0 0 0 000 0 0 000 01 00 0 1 0 0 00 0000 00000 0 000 00 000 00 0 00 00 000011

0.0 E. Roquain

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

15 / 33

0 Doing like for 1 test? t ≡ α = 0.1 0

0.20

0 0 0 0 0

0.15

0 0000

0 000 0

0.00

0.05

0.10

0 0 0 0 0 0 0

E. Roquain

0 00 10 1 000 0

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

15 / 33

0 Union bound Bonferroni? t ≡ α/m = 0.1/100 0

0.20

0 0 0 0 0

0.15

0 0000

0 000 0

0.00

0.05

0.10

0 0 0 0 0 0 0

E. Roquain

0 00 10 1 000 0

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

16 / 33

0.20

Union bound Bonferroni? t ≡ α/m = 0.1/100 1 0 0 0

0.15

0 0 11 0

0.00

0.05

0.10

00 1 01 0 1 011

E. Roquain

111 1 1 01 1 1 11 01 1110 1 11 11 1101 11 11111111111111

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

16 / 33

0.00

0.05

0.10

0.15

0.20

Do something in between! t` = α`/m = 0.1`/100

E. Roquain

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

17 / 33

0.20

Do something in between! t` = α`/m = 0.1`/100 1 0 0 0

0.15

0 0 11 0

0.00

0.05

0.10

00 1 01 0 1 011

E. Roquain

111 1 1 01 1 1 11 01 1110 1 11 11 1101 11 11111111111111

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

17 / 33

0 Do something in between! t` = α`/m = 0.1`/100 0

0.20

0 0 0 0 0

0.15

0 0000

0 000 0

0.00

0.05

0.10

0 0 0 0 0 0 0

E. Roquain

0 00 10 1 000 0

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 Introduction

17 / 33

Smart ! . . . and rigorous ?

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

Introduction

18 / 33

1 Introduction 2 False discovery rate control 3 FDR in other statistical issues

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

False discovery rate control

19 / 33

BH procedure p-value view

c.d.f. view

kb = max{0 ≤ i ≤ m : p(i) ≤ αi/m}

b m (t) ≥ t/α} bt = max{t ∈ [0, 1] : G

bt = αkb/m b i = 1{pi ≤ bt} = 1{Xi ≥ Φ−1 (bt)} H E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

False discovery rate control

20 / 33

False discovery rate control

b i = 1{pi ≤ bt} (∀i), For a decision H b i = 1} #{i : Hi = 0, H FDP(bt) = b i = 1} #{i : H FDR(bt) = E[FDP(bt)]



0 =0 0



Theorem [Benjamini and Hochberg (1995)] [Benjamini and Yekutieli (2001)] If Γ = Im and bt threshold of BH procedure, ∀µ, H, FDR(bt) = (m0 /m)α ≤ α

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

False discovery rate control

21 / 33

Simulations. m = 50; m0 = 25; µ = 3; 0 Γ = Im FDP(BH) = 0

00 0 0

0.3

0.4

0

0.0

0.1

0.2

00 000 0 1 0

E. Roquain

1 1 1 111111111111111111111 0.0 0.2 0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

0.0

0.1

0.2

0.3

0.4

0 Γ=I Simulations. m = 50; m0 = 25; µ = 3; m

E. Roquain

FDP(BH) = 0.16

0 00 0 0

0 0 0 11 1 0 0 110 1110 0 1 1 1 1 1 1 1 1 1 1 111111 0.0 0.2 0.4 0.6 FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

0.0

0.1

0.2

0.3

0.4

Simulations. m = 50; m0 = 25; µ = 03; Γ = Im

E. Roquain

00

FDP(BH) = 0.0833

00 0 1 0

01 10 0 1 11 1 11 11 1 1 1 0 1 0 1 1 1 1111111 0.0 0.2 0.4 0.6 FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

0.0

0.1

0.2

0.3

0.4

Simulations. m = 50; m0 = 25; µ 0= 3; Γ = Im

E. Roquain

FDP(BH) = 0.08

1 010 0 0 0 1 1 1 1 00 1 111111111111111111 0.0 0.2 0.4 0.6 FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

0.0

0.1

0.2

0.3

0.4

Simulations. m = 50; m0 = 25; µ = 3; Γ = Im

E. Roquain

00 00

FDP(BH) = 0.12

0 1 0 0 1 0 10

0 101 101 11 1 1111 111111011111 0.0 0.2 0.4 0.6 FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

01 0 0 0 00

FDP(BH) = 0.167

0.3

0.4

Simulations. m = 50; m0 = 25; µ = 3; Γ = Im

1 0 1 1 1

0.0

0.1

0.2

0

E. Roquain

00 1 1 0 0111 111111111111111 0.0 0.2 0.4 0.6 FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

0 FDP(BH) = 0.0435

0 0 1 10

0.0

0.1

0.2

0.3

0.4

Simulations. m = 50; m0 = 25; µ0= 3; Γ = Im

E. Roquain

0 1 10 1 1 1 1 1 111 1111111111111 0.0 0.2 0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

FDP(BH) = 0

00 0

0.0

0.1

0.2

0.3

0.4

Simulations. m = 50; m0 = 25; µ00= 3; Γ = Im

E. Roquain

1 0 11 1 00 11 1111 1 1 1 11 1111111111 0.0 0.2 0.4 0.6 FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

0 0 00 0 10

FDP(BH) = 0

0.0

0.1

0.2

0.3

0.4

Simulations. m = 50; m0 = 25; µ = 3;00Γ = Im

E. Roquain

0 0 010 0 01 0 1 0 1 11 1 1 1111111111111111 0.0 0.2 0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

00 0 0 0 0 0

FDP(BH) = 0.04

0.3

0.4

0 Γ=I Simulations. m = 50; m0 = 25; µ =003; m

0.2

0

0.0

0.1

1

E. Roquain

0 1 111 0111 11111111111111111 0.0 0.2 0.4 0.6 FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 False discovery rate control

22 / 33

800 600 400 0

200

Number of citations

1000

1200

sional application areas, such as neural imaging, are on the rise also, showing its ability to be applied in many diverse types of application. The list of the 10 highest cited papers that cite Benjamini and Hochberg (1995), which isand shown in Table(1995). 1, is particularly interesting, it includes statistical suggesting Benjamini Hochberg Controlling the falsebecause discovery rate: a six practical andpapers, powerful approach to that further theoretical and methodological developments of the method have had significant influence. multiple testing

1996

1998

2000

2002

2004

2006

2008

Year

Fig. 1. Rapidly increasing number of citations of Benjamini and Hochberg (1995), suggesting that its influ[Benjamini (2010,JRSSB)] ence has not yet reached its peak (note that the figure for 2009 is only partially shown) 10 most cited papers that Benjamini and now > Table 20, 1.000 citations oncitegoogle scholar ! Hochberg (1995)

E. Roquain

Rank

Article citing Benjamini

Number of

FDR : introduction, enjeux et perspectives. Part I.

False discovery rate control

23 / 33

1 Introduction 2 False discovery rate control 3 FDR in other statistical issues

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

24 / 33

0.00

0.05

0.10

0.15

0.20

Why should FDR thresholding be adaptive to sparsity?

E. Roquain

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 FDR in other statistical issues

25 / 33

0 Why should FDR thresholding be adaptive to sparsity? 0

0.20

0 0 0 0 0

0.15

0 0000

0 000 0

0.00

0.05

0.10

0 0 0 0 0 0 0

E. Roquain

0 00 10 1 000 0

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 FDR in other statistical issues

25 / 33

0.20

Why should FDR thresholding be adaptive to sparsity? 1 0 0 0

0.15

0 0 11 0

0.00

0.05

0.10

00 1 01 0 1 011

E. Roquain

111 1 1 01 11 11 1 0 1110 11 11 1 1101 11 11111111111111

0.0

0.2

0.4

0.6

FDR : introduction, enjeux et perspectives. Part I.

0.8

1.0 FDR in other statistical issues

25 / 33

[Linnemann] - increasing signal strength

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

26 / 33

[Linnemann] - increasing signal strength

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

26 / 33

[Linnemann] - increasing signal strength

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

26 / 33

[Linnemann] - increasing signal strength

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

26 / 33

[Linnemann] - increasing signal strength

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

26 / 33

Adaptation to unknown sparsity

bt seems "adaptive" to the “quantity" of signal in the data I Classification : where is the signal ? [Bogdan et al. (2011)], [Neuvial and R. (2012)] I Detection: is there some signal ? [Ingster (2002)], [Donoho and Jin (2004)], etc I Estimation: what is the value EX of the signal ? d = Xi 1{|Xi | ≥ bt} (hard thresholding) EX [Abramovich et al. (2006)], [Donoho and Jin (2006)]

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

27 / 33

Classification Xi ∼ π0,m N (0, 1) + (1 − π0,m ) N (µm , 1), 1 ≤ i ≤ m, i.i.d. but π0,m → 1 (sparse) and µm → ∞ (compensates sparsity). I training set = null distribution known (one-class classification) bm : R → {0, 1}; I Classification rule h I Risk   m X −1 −1 ˆ ˆ Rm (hm ) = (1 − π0 ) E m 1{hm (Xi ) 6= Hi } . i=1

I Classification boundary in (sparsity, signal) space such that ˆm : Rm (h ˆm ) → 0 (perfect classification) Above the boundary, ∃h ˆ m , Rm ( h ˆm ) → 1 (unclassifiable) Under the boundary, ∀h

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

28 / 33

1.0

Classification boundary

µm =

0.8

Perfect classification

p 2r log m

π0,m = 1 − m−β

0.6

bBH (x) = 1{x ≥ Φ−1 (bt)} BH h m

0.4

r

with αm ∝ (log m)−1/2 I Classification boundary attained by BH.

0.2

Unclassifiable

0.0

On the boundary : risk BH ∼ Bayes risk. 0.5

0.6

0.7

0.8

0.9

1.0

β

[Bogdan et al. (2011)], [Neuvial and R. (2012)]

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

29 / 33

Detection : is there some signal ? Same model Xi ∼ π0,m N (0, 1) + (1 − π0,m ) N (µm , 1), 1 ≤ i ≤ m, i.i.d. but π0,m → 1 (sparse) and µm → ∞ (compensates sparsity). I Test H0 : “N (0, Im )" against H1 : “mixture". I Risk Rm (T ) = PH0 (T (X ) = 1) + PH1 (T (X ) = 0) I Detection boundary in (sparsity, signal) space such that Above the boundary, ∃T : Rm (T ) → 0 (perfect detection) Under the boundary, ∀T , Rm (T ) → 1 (undetectable)

E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

30 / 33

Detection boundary 1.0

µm =

2r log m

π0,m = 1 − m−β

Perfect classification

0.8

p

T BH = 1{∃i : p(i) ≤ αm i/m}

0.6

with αm ∝ (log m)−1/2 I Detection boundary attained by BH when β ∈ (3/4, 1)

0.2

0.4

r

Perfect detection

0.0

Undetectable

0.5

0.6

0.7

0.8 β

E. Roquain

0.9

1.0

I Better to use “higher criticism" ( ) √ i/m − p(i) max mp i p(i) (1 − p(i) )

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

31 / 33

LASSO and FDR Regression with orthogonal design: X ∼ N (β, Im ) [Bogdan et al. (2013)]: sorted `1 penalized estimator (SLOPE) ) ( m X 1 2 b β = arg minm ||X − β|| + λk |β|(k ) β∈R 2 k =1

where λ1 ≥ λ2 ≥ · · · ≥ λm ; |β|(1) ≥ |β|(2) ≥ · · · ≥ |β|(m) Selection with {i : βbi = 6 0}: −1

p (α/(2m)) ' 2 log m p −1 I λk = Φ (αk /(2m)) ' 2 log(m/k ) I λk = λ = Φ

E. Roquain

Bonferroni BH !

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

32 / 33

LASSO and FDR Regression with orthogonal design: X ∼ N (β, Im ) [Bogdan et al. (2013)]: sorted `1 penalized estimator (SLOPE) ) ( m X 1 2 b β = arg minm ||X − β|| + λk |β|(k ) β∈R 2 k =1

where λ1 ≥ λ2 ≥ · · · ≥ λm ; |β|(1) ≥ |β|(2) ≥ · · · ≥ |β|(m) Selection with {i : βbi = 6 0}: −1

p (α/(2m)) ' 2 log m p −1 I λk = Φ (αk /(2m)) ' 2 log(m/k )

I λk = λ = Φ

E. Roquain

Bonferroni BH !

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

32 / 33

Outlook Some conclusions for FDR ⊕ Very simple ⊕ Trade-off type I / power ⊕ Adaptive to sparsity

Some issues ! Sensitive to null hypothesis ! Choosing α ! Calibrating test statistics

Main challenge What about dependence ? E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

33 / 33

Outlook Some conclusions for FDR ⊕ Very simple ⊕ Trade-off type I / power ⊕ Adaptive to sparsity

Some issues ! Sensitive to null hypothesis ! Choosing α ! Calibrating test statistics

Main challenge What about dependence ? E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

33 / 33

Outlook Some conclusions for FDR ⊕ Very simple ⊕ Trade-off type I / power ⊕ Adaptive to sparsity

Some issues ! Sensitive to null hypothesis ! Choosing α ! Calibrating test statistics

Main challenge What about dependence ? E. Roquain

FDR : introduction, enjeux et perspectives. Part I.

FDR in other statistical issues

33 / 33