Technical appendix to “Adaptive estimation of stationary Gaussian fields” Nicolas Verzelen Universit´ e Paris Sud, Laboratoire de Math´ ematiques, UMR 8628 Orsay Cedex F-91405 e-mail: [email protected] Abstract This is a technical appendix to “Adaptive estimation of stationary Gaussian fields” [6]. We present several proofs that have been skipped in the main paper. These proofs are organised as in Section 8 of [6]. AMS 2000 subject classifications: Primary 62H11; secondary 62M40. Keywords and phrases: Gaussian field, Gaussian Markov random field, model selection, pseudolikelihood, oracle inequalities, Minimax rate of estimation.

1. Proof of Proposition 8.1 Proof of Proposition 8.1. First, we recall the notations introduced in [3]. Let N be a positive integer. Then, IN stands for the family of subsets of {1, . . . , N } of size less than 2. Let T be a set of vectors indexed by IN . In the sequel, T is assumed to be a compact subset of R(N (N +1)/2)+1 . The following lemma states a slightly modified version of the upper bound in remark 7 in [3]. Lemma 1.1. Let T be a supremum of Rademacher chaos indexed by IN of the form X N X T := sup t{i} + t∅ , Ui Uj t{i,j} + t∈T

i=1

{i,j}

where U1 , . . . , UN are independent Rademacher random variables. Then for any x > 0,   x x2 ∧ , (1) P {T ≥ E[T ] + x} ≤ 4 exp − L1 E[D]2 L2 E where D and E are defined by: D

:=

sup

sup

t∈T α:kαk2 ≤1

E

:=

sup

X X N U α t i j {i,j} , i=1

j6=i

sup

t∈T α(1) ,α(2) ,kα(1) k2 ≤1 kα(2) k≤1

1

X N X (1) (2) t α α {i,j} i j . i=1 j6=i

Verzelen/Technical appendix

2

Contrary to the original result of [3], the chaos are not assumed to be homogeneous. Besides, the t{i} are redundant with t∅ . In fact, we introduced this family in order to emphasize the connection with Gaussian chaos in the next result. A suitable application of the central limit theorem enables to obtain a corresponding bound for Gaussian chaos of order 2. Lemma 1.2. Let T be a supremum of Gaussian chaos of order 2. X X 2 T := sup t{i,j} Yi Yj + ti Yi + t∅ , t∈T

(2)

i

{i,j}

where Y1 , . . . , YN are independent standard Gaussian random variable. Then, for any x > 0,   x x2 ∧ , (3) P {T ≥ E[T ] + x} ≤ exp − E[D]2 L1 EL2 where D

:=

sup

X

sup

Yi (1 + δi,j )αj t{i,j} ,

t∈T α∈RN kαk2 ≤1 i,j

E

:=

sup

sup

X

sup

α1,i α2,j t{i,j} (1 + δi,j ) .

t∈T α1 , kα1 k2 ≤1 α2 , kα2 k2 ≤1 i,j

The proof of this Lemma is postponed to the end of this section. To conclude, we derive the result of Proposition 8.1 from this last lemma. For any matrix R ∈ F , we define the vector tR ∈ Rnr(nr+1)/2+1 indexed by Inr as follows tR {(i,k),(j,l)} := δk,l (2 − δi,j )

R[i,j] , n

tR {(i,k)} :=

R[i,i] , n

and tR ∅ := −tr(R) ,

where δi,j is the indicator function of i = j. In order to apply Lemma 1.2 with  N = nr and T = tR |R ∈ F , we have to work out the quantities D and E.

D

=

sup

sup

 n r X X

Y [i,k]

tR ∈T α∈Rnr , kαk2 ≤1  i=1 k=1

=

sup

sup

2

 r X n X

r X n X

tR,k,l (1 + δi,j δk,l )αjl ij



j=1 l=1

 r X R[i,j]αk  j

Y [i,k]  n  i=1 k=1 j=1  ! n r r  2 X X k X = sup sup αj Y [i,k]R[i,j] .  R∈F α∈Rnr , kαk2 ≤1 n  j=1 i=1 R∈F α∈Rnr , kαk2 ≤1

k=1

 

Verzelen/Technical appendix

3

Applying Cauchy-Schwarz identity yields  !2  n X r r X  X 4 sup D2 = Y [i,k]R[i,j]  n2 R∈F  k=1 j=1

=

i=1

4 sup tr(RY Y ∗ R∗ ) . n R∈F

(4)

Let us now turn the constant E E

=

sup tR ∈T

=

sup R∈F

sup α1 , α2 ∈ Rnr kα1 k2 ≤ 1, kα2 k2 ≤ 1

X

X

k l (1 + δij δk,l )tR,kl i,j α1,i α2,j

1≤i,j≤r 1≤k,l≤n

2 X X k k R[i,j]α1,i α2,j . sup n nr 1≤i,j≤r 1≤k≤n α1 , α2 ∈ R kα1 k2 ≤ 1, kα2 k2 ≤ 1

From this last expression, it follows that E is a supremum of L2 operator norms   2 E = sup ϕmax Diag (n) (R) , n R∈F where Diag (n) (R) is the (nr×nr) block diagonal matrix such that each diagonal block is made of the matrix R. Since the largest eigenvalue of Diag (n) (R) is exactly the largest eigenvalue of R, we get E=

2 sup ϕmax (R) . n R∈F

(5)

Applying Proposition 1.2 and gathering identities (4) and (5) yields   ^ t  t2 , P(Z ≥ E(Z) + t) ≤ exp − L1 E(V ) L2 B where B = E and V = D2 .

Proof of Lemma 1.1. This result is an extension of Corollary 4 in [3]. We shall closely follow the sketch of their proof adapting a few arguments. First, we upper bound the moments of (T − E(T ))+ . Then, we derive the deviation inequality from it. Here, x+ = max(x, 0). Lemma 1.3. For all real numbers q ≥ 2, p k(T − E(T ))+ kq ≤ LqE(D) + LqE ,

(6)

where kT kqq stands for the q-th moment of the random variable T . The quantities D and E are defined in Lemma 1.1.

Verzelen/Technical appendix

4

By Lemma 1.3, for any t ≥ 0 and any q ≥ 2,   E (T − E(T ))q+ P (T ≥ E(T ) + t) ≤ tq √ q LqE(D) + LqE ≤ . t √ The right-hand side is at most 2−q if LqE(D) ≤ t/4 and LqE ≤ t/4. Let us set t t2 ∧ . q0 := 16LE(D)2 4LE If q0 ≥ 2, then P (T ≥ E(T ) + t) ≤ 2−q0 . On the other hand if q0 < 2, then 4 × 2−q0 ≥ 1. It follows that    log(2) t t2 P (T ≥ E(T ) + t) ≤ 4 exp − ∧ . 4L 4E(D)2 E

Proof of Lemma 1.3. This result is based on the entropy method developed in [3]. Let f : RN → R be a measurable function such that T = f (U1 , . . . , UN ). In 0 denote independent copies of U1 , . . . , UN . The random the sequel, U10 , . . . , UN variable Ti0 and V + are defined by Ti0 V+

:= f (U1 , . . . , Ui−1 , Ui0 , Ui+1 , . . . , UN ) , "N # X 0 2 N := E (T − Ti )+ |U1 , i=1

where U1N refers to the set {U1 , . . . , UN }. Theorem 2 in [3] states that for any real q ≥ 2, √ p k(T − E(T ))+ kq ≤ Lqk V + kq . (7) √ To conclude, we only have bound the moments of V + . By definition, X N X T = sup Ui Uj t{i,j} + t{i} + t∅ . t∈T

i=1

{i,j}

Since the set T is compact, this supremum is achieved almost surely at an element t0 of T . For any 1 ≤ i ≤ N , X 2  0 2 0 0 (T − Ti )+ ≤ (Ui − Ui ) Uj t {i, j} . j6=i

Verzelen/Technical appendix

5

Gathering this bound for any i between 1 and N , we get   X 2  N X V+ ≤ Uj t0 {i, j} U1N  E  (Ui − Ui0 ) i=1 j6=i 2 N X X ≤ 2 Uj t0 {i, j} i=1

≤

2

j6=i

X N

sup α∈RN , kαk2 ≤1

≤

2 sup t∈T

X

i=1

t0{i,j} Uj

2

j6=i

N  X

sup α∈RN ,

αi

Ui

kαk2 ≤1 i=1

X

2 αj t{i,j}

= 2D2 .

j6=i

Combining this last bound with (7) yields p √ Lq 2kDkq k(T − E(T ))+ kq ≤ i p h ≤ Lq E(D) + |(D − E(D))+ kq .

(8)

Since the random variable D defined in Lemma 1.1 is a measurable function f2 of the variables U1 , . . . , UN , we apply again Theorem 2 in [3].

p q +

k(D − E(D))+ kq ≤ Lq V2

, q

where

V2+

is defined by " V2+ := E

N X i=1

# (D − Di0 )2+ UiN ,

and Di0 := f2 (U1 , . . . , Ui−1 , Ui0 , Ui+1 , . . . , UN ). As previously, the supremum in D is achieved at some random parameter (t0 , α0 ). We therefore upper bound V2+ as previously.    X 2 N X U1N  V2+ ≤ E  (Ui − Ui0 ) αj0 t0{i,j} i=1

≤ 2

i=1

≤ 2

j6=i

N X X

αj0 t0{i,j}

2

j6=i

X N

sup α(2) ∈RN ,kαk

2 ≤1

i=1

(2) αj

X

αi0 t{i,j}

2

j6=i

Gathering this upper bound with (8) yields p k(T − E(T ))+ kq ≤ LqE(D) + LqE .

= 2E 2 .

Verzelen/Technical appendix

6

Proof of Lemma 1.2. We shall apply the central limit theorem in order to transfer results for Rademacher chaos to Gaussian chaos. Let f be the unique function satisfying T = f (y1 , . . . , yN ) for any (y1 , . . . , yN ) ∈ RN . As the set T is com(j) pact, the function f is known to be continuous. Let (Ui )1≤i≤N,j≥0 an i.i.d. family of Rademacher variables. For any integer n > 0, the random variables Y (n) and T (n) are defined by Y (n) T (n)

n (j)  (j) X UN U1 √ ,..., √ , n n j=1 j=1   := f Y (n) .

X n

:=

Clearly, T (n) is a supremum of Rademacher chaos of order 2 with nN variables and a constant term. By the central limit theorem, T (n) converges in distribution towards T as n tends to infinity. Consequently, deviation  inequalities for the variables T (n) transfer to T as long as the quantities E D(n) , E (n) , and E[T (n) ] converge. We first prove that the sequence T (n) converges in expectation towards T . As T (n) converges in distribution, it is sufficient to show that the sequence T (n) is asymptotically uniformly integrable. The set T is compact, thus there exists a positive number t∞ such that X  (n) (n) T (n) ≤ t∞ |Yi Yj | + 1 i,j

 N  2  X (n) ≤ t∞ 1 + (N + 1)/2 Yi . i=1

It follows that 

T

(n)

2

≤

t2∞



N +1 2

2

" # N  4 X N +2 (n) 1+ Yi . 2 i=1

(9)

(n)

The sequence Yi does not only converge in distribution to a standard normal distribution but also in moments (see for instance [1] p.391). It follows that h
2 i
limE T (n) ≤ ∞ and the sequence f Y (n) is asymptotically uniformly integrable. As a consequence, h i lim E T (n) = E[T ] . n→∞

  Let us turn to the limit of E D(n) . As the variable T (n) equals T

(n)

X = sup t{i,j} t∈T {i,j}

(k)

X 1≤k,l≤n

(l)

Ui Uj n

+

X i

ti

X X U (k) X U (l) i i √ √ + t∅ + ti , n n i

1≤k≤n

l6=k

Verzelen/Technical appendix

7

it follows that D

(n)

X

X

X t{i} (l)  t{i,j} X (l) = sup sup αj + 2 2 α n n i t∈T α∈RnN , kαk2 ≤1 1≤i≤N 1≤k≤n 1≤l≤n l6=k j6=i P  X (k) X (l)  Ui 1≤l≤n αj √ √ ≤ sup sup (1 + δi,j )t{i,j} + A(n) , (10) n n t∈T α∈RnN , kαk2 ≤1 i j X

(k) Ui

where the random variable A(n) is defined by A(n) := sup

N X n X

sup

(j)

t{i}

t∈T α∈RnN , kαk2 ≤1 i=1 j=1

Straightforwardly, one upper bounds A

(n)

by t∞ /n

Ui αj . n i

r PN Pn i=1

j=1



(j)

Ui

2

and

its expectation satisfies r   N , E A(n) ≤ t∞ n which goes to 0 when n goes to infinity. Thus, we only have to upper bound the expectation of the first term in (10). Clearly, the supremum is achieved only (l) when for all 1 ≤ j ≤ N , the sequence (αj )1≤l≤n is constant. In such a case, √ (1) the sequence (αj )1≤j≤N satisfies kα(1) k2 ≤ 1/ n. it follows that  h i (n) E D = E sup t∈T

 X   X 1 (n) . E Yi (1 + δi,j )αj sup +O √ n α∈RN kαk2 ≤1 i j

Let g be the function defined by g (y1 , . . . , yN ) = sup

X

sup

t∈T α∈RN kαk2 ≤1

yi

 X (1 + δi,j )αj ,

i

j

for any (y1 , . . . , yN ) ∈ RN . The function g(.) is measurable and continuous as the supremum is taken over a compact set. As a consequence, g(Y (n) ) converges in distribution towards g(Y ). As previously, the sequence is asymptotically uniformly integrable since its moment of order 2 is uniformly upper bounded. It follows that lim E D(n) = E [D]. Third, we compute the limit of E (n) . By definition, E

(n)

=

sup

sup

N X n X

t∈T α1 ,α2 ∈RnN , kα1 k2 ≤1,kα2 k2 ≤1 i=1 k=1

=

sup t∈T α1 ,α2 ,

N X N X

k α1,i

XX n

(l) t{i,j} α2,j

j6=i l=1

"

n

+2

X l6=k

#

(l) t{i} α2,i



n

  n n t{i,j} X X (k) (l) 1 sup (1 + δi,j ) α1,i α2,j + O . n n kα1 k2 ≤1,kα2 k2 ≤1 i=1 j=1 k=1 l=1

Verzelen/Technical appendix

8

As for the computation of D(n) , the supremum is achieved when the sequences k l (α1,i )1≤k≤n and (α2,j )1≤l≤n are constant for any i ∈ {1, . . . , N }. Thus, we only have to consider the supremum over the vectors α1 and α2 in RN . E (n) = sup

sup

N X N X

t∈T α1 ,α2 ∈RN kαi k2 ≤1 i=1 j=1

(1 + δij )ti,j α1,i α2,j + O

  1 . n

It follows that E (n) converges towards E when n tends to infinity. The random variable T (n) −E(T (n) ) converges in distribution towards T −E(T ). By Lemma 1.1 ,   x2 x ∧ , P(T − E(T ) ≥ x) ≤ lim exp − E[D(n) ]2 L1 E (n) L2 for any x > 0. Combining this upper bound with the convergence of the sequences D(n) and E (n) allows to conclude.

2. Proof of Theorem 3.1 Proof of Lemma 8.3. We only consider here the anisotropic case, since the isotropic case is analogous. This result is based on the deviation inequality for suprema of Gaussian chaos of order 2 stated in Proposition 8.1. For any model m0 belonging to M, we shall upper bound the quantities E(Zm0 ), Bm0 , and E(Wm0 ) defined in (42) in [6]. 0 1. Let us first consider the expectation of Zm0 . Let Um,m 0 be the new vector space defined by √ DΣ 0 Um,m , 0 := Um,m0 p

where Um,m0 is introduced in the proof of Lemma 8.2 in [6]. This new space allows to handle the computation with the canonical inner product in the (2) 0 space of matrices. Let Bm2 ,m02 be the unit ball of Um,m 0 with respect to √ the canonical inner product. If R belongs to Um,m0 , then kRkH0 = kR DΣ /pkF , where k.kF stands for the Frobenius norm. Zm0

=

 1  tr RDΣ (YY∗ − Ip2 ) 2 p  √ 
DΣ ∗ sup tr R YY − Ip2 p (2) R∈B 2 02 m ,m

√


DΣ ∗

ΠU 0 YY − Ip2

m,m0 p

, F sup

0 R∈BH2 02 m ,m

=

=

(11)

Verzelen/Technical appendix

9

where ΠU 0 0 refers to the orthogonal projection with respect to the m,m 0 canonical inner product onto the space Um,m de0 . Let F1 , . . . , Fd 2 m ,m02 0 note an orthonormal basis of Um,m . 0 dm2 ,m02 2 E(Zm 0)

=

X

 E tr2

s Fi

i=1 dm2 ,m02

=

X i=1

X i=1 dm2 ,m02

≤

!

√ X 2 p2 DΣ [j,j] ∗ E Fi [j,j] (YY [j,j] − 1) p j=1

dm2 ,m02

=


DΣ YY∗ − Ip2 2 p

X i=1

2 tr(Fi DΣ Fi ) np2 2dm2 ,m02 ϕmax (Σ) 2ϕmax (DΣ ) = . np2 np2

Applying Cauchy-Schwarz inequality, it follows that s 2dm2 ,m02 ϕmax (Σ) E(Zm0 ) ≤ . np2

(12)

2. Using the identity (11), the quantity Bm0 equals   √ 2 DΣ Bm 0 = sup ϕmax R . n R∈B(2) p m2 ,m02

As the operator norm is under-multiplicative and as it dominates the Frobenius norm, we get the following bound p 2 ϕmax (Σ) Bm0 ≤ . (13) np 3. Let us turn to bounding the quantity E(Wm0 ). Again, by introducing the (2) ball Bm2 ,m02 , we get Wm 0

=

4 sup n R∈BH0

 1  tr RYY∗ DΣ R 2 p

m2 ,m02

≤

≤

  4ϕmax (Σ) sup tr RYY∗ R 2 np (2) R∈B 2 02 m ,m   
 4ϕmax (Σ) ∗ 1+ sup tr R YY − Ip2 R . np2 (2) R∈B m2 ,m02

Verzelen/Technical appendix

10

0 Let F1 , . . . Fdm2 ,m02 an orthonormal basis of Um,m 0 and let λ be a vector dm2 ,m02 in R . We write kλk2 for its L2 norm.   
2 tr R YY∗ − Ip2 R E sup R∈B

(2) m2 ,m02

 = E sup

dm2 ,m02

X



λi λj tr Fi Fj

(YY∗ /n

− Ip2 )



2

kλk2 ≤1 i,j=1 dm2 ,m02

X

≤

  2  . E tr Fi Fj (YY∗ /n − Ip2 )

i,j=1

The second inequality is a consequence of Cauchy-Schwarz inequality in 2 d2 R(dm2 ,m02 ) since the l2 norm of the vector (λi λj )1≤i,j≤dm2 ,m02 ∈ R m2 ,m02 is bounded by 1. Since the matrices Fi are diagonal, we get  E

sup R∈B

∗

tr [R(YY /n − I)R]

2



(2) m2 ,m02

2 ≤ n

dm2 ,m02

X

kFi Fj k22 .

i,j=1

It remains to bound the norm of the products Fi Fj for any i, j between 1 and dm2 ,m02 .  2 dm2 ,m02 dm2 ,m02 p2 dm2 ,m02 p2 X X X X X  kFi Fj k22 = Fi [k,k]2 Fj [k,k]2 = Fi [k,k]2  . i,j=1

i,j=1 k=1

k=1

i=1

Pdm2 ,m02 For any k ∈ {1, . . . , p2 }, Fi [k,k]2 ≤ 1 since (F1 , . . . , Fdm2 ,m02 ) i=1 form an orthonormal family. Hence, we get dm2 ,m02

X

2

kFi Fj k22

≤

i,j=1

d

p m2 ,m02 X X k=1

Fi [k,k]2 = dm2 ,m02 .

i=1

All in all, we have proved that " # r 2dm2 ,m02 4ϕmax (Σ) E(Wm0 ) ≤ 1+ . np2 n

(14)

Gathering these three bounds and applying Proposition 8.1 allows to obtain the following deviation inequality:  q np o p (Σ) 2 02 P Zm0 ≥ 2ϕmax 1 + α/2 d + ξ m ,m n ( " √
√ 2 √  √
√  #) V n 1+α/2−1 dm2 ,m02 +ξ 1+α/2−1 dm2 ,m02 +ξ √ √ ≤ exp − 2L2 2L1 (1+ 2dm2 ,m02 /n)      √ 2 V nωm,m0 V √nξ ωm,m0 ξωm,m0 √ √ ≤ exp − 2L 1+√2d − L 1+√2d , 2L2 2L2 1( 1[ m2 ,m02 /n) m2 ,m02 /n]

Verzelen/Technical appendix

where ωm,m0 =

11

p p 1 + α/2 − 1 dm2 ,m02 . As n and dm2 ,m02 are larger than

one, there exists a universal constant L02 such that " p # p √ p ( 1 + α/2 − 1)2 dm2 ,m02 ^ n( 1 + α/2 − 1) dm2 ,m02 p √
2L1 1 + 2dm2 ,m02 /n 2L2  2 p  p p 0 1 + α/2 − 1 ∧ 1 + α/2 − 1 . ≥ 4L2 dm2 ,m02 Since the vector space Um,m0 contains all the matrices D(θ0 ) with θ0 belonging to m0 , dm2 ,m02 isp larger than dm0 . Besides, by concavity of the square root function, p √ it√holds that 1 + α/2 − 1 ≥ α[4 1 + α/2]−1 . Setting L01 := [4L1 (1 + 2)]−1 ∧ [ 2L2 ]−1 and arguing as previously leads to " # p p √ √ ξ( 1 + α/2 − 1) dm2 ,m02 ^ nξ α 0 p √
≥ L1 ξ p ∧ n . L1 1 + 2dm2 ,m02 /n 2L2 1 + α/2 Gathering these two inequalities allows us to conclude that ! r o n 2ϕmax (Σ) p P

Z m0 ≥

(1 + α/2) dm2 ,m02 + ξ

n

( ≤

exp

−L02

p

dm0

α2 p ∧ 1 + α/2 1 + α/2 α

!

" −

L01 ξ

α

p

#)

√ ∧ n

.

1 + α/2

Proof of Lemma 8.4 in [6]. The approach falls in two parts. First, we relate the dimensions dm and dm2 to the number of nodes of the torus Λ that are closer than rm or 2rm to the origin (0, 0). We recall that the quantity rm is introduced in Definition 2.1 of [6]. Second, we compute a nonasymptotic upper bound of the number of points in Z2 that lie in the disc of radius r. This second step is quite tedious and will only give the main arguments. Let m be a model of the collection M1 . By definition, m is the set of points lying in the disc of radius rm centered on (0, 0). Hence, Θm = vect {Ψi,j , (i, j) ∈ m} , where the matrices Ψi,j are defined by Eq. (14) in [6]. As Ψi,j = Ψ−i,−j , the dimension dm of Θm is exactly the number of orbits of m under the action of the central symmetry s. As dm2 is defined as the dimension of the space Um , it also corresponds to the dimension of the space  vect {C(θ), θ ∈ Θm } + vect C(θ)2 , θ ∈ Θm , (15)