Normalized Convolution Interpolated from an Adaptive Subgrid Benjamin Ménétrier

NCAR - Boulder, Colorado August 1st , 2017

Principles

Grids

Outline Principles Subgrid denition Convolution function MPI communications Applying NICAS Conclusions 1

Convolution

MPI

Application

Conclusions

Principles

Grids

Outline Principles Subgrid denition Convolution function MPI communications Applying NICAS Conclusions 2

Convolution

MPI

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

Conclusions

Explicit convolution We want to apply a correlation matrix on any grid type. Advantages of an explicit convolution C ∈ Rn×n : • Explicite choice of the convolution function • Exact normalization (C = 1) ii • Work on any grid type! Drawback: cost scales as O (n2 )...

e = SCs ST where To limit the cost, work on a grid subset: C n×ns • S∈R is an interpolation from the subgrid s ns ×ns • C ∈R is a convolution matrix on the subgrid If ns  n, then the cost scales as O (n) (interpolation).

3

Problems with working on a subgrid: • If ns is too small, the fonction is distorded, • Normalization breaks down because of the interpolation: even e is not. if Cs is normalized, C

Principles

Grids

Convolution

MPI

Application

Convolution explicite NICAS method (Normalized Interpolated Convolution from an Adaptive Subgrid) :

Ce = NSCs ST NT

where N is a diagonal normalization matrix. Several questions: • What subgrid? What interpolation? • What convolution function? • How to compute the normalization? • What parallelization method? • What results/cost/scalability? • ...

4

Conclusions

Principles

Grids

Outline Principles Subgrid denition Convolution function MPI communications Applying NICAS Conclusions 5

Convolution

MPI

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

Subgrid choice Full 3D interpolation: hard to compute and to apply in a geophysical model ⇒ S split into three operators:

S = Sh Sv Ss

Grid

G

s

Subset of points (number of points) c

c

2

2

Subset of levels (number of levels)

S (n )

v

c

c

1

1

S (n )

Vertical interpolation G

h

c

c

1

1

6

f

c

c

0

0

S

S

S (n )

S (n )

c

S depending

s

2

c

c

2

1

from S to S

l

l

1

1

S (n )

Horizontal interpolation G

l

1

S (n )

Horizontal interpolation G

l

1

Comment

v

l

l

0

l

l

1

0

S (n ) S

h l

l

0

0

c

1

l

l

1

0

n ≤n c

c

1

0

from S to S

S (n )

c

2

n ≤n

from S to S

0

on the level

c

c

1

0

n ≤n

Conclusions

Principles

Grids

Convolution

MPI

Application

Choice of the horizontal grid

c

Black dots: ARPEGE grid at truncation TL149 (set S ) 0

7

Conclusions

Principles

Grids

Convolution

MPI

Choice of the horizontal grid

Green dots: basic subset S 7

c

1

Application

Conclusions

Principles

Grids

Convolution

MPI

Choice of the horizontal grid

c

Red dots: nal subset S 2 at a level with a short support radius 7

Application

Conclusions

Principles

Grids

Convolution

MPI

Choice of the horizontal grid

c

Red dots: nal subset S 2 at a level with a medium support radius 7

Application

Conclusions

Principles

Grids

Convolution

MPI

Choice of the horizontal grid

c

Red dots: nal subset S 2 at a level with a large support radius 7

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

Choice of the vertical grid Levels sampled depending on the vertical support radius. Example: uniform log(pressure) vertical support radius

8

ARPEGE levels (black dots) / subgrid levels (red dots)

Conclusions

Principles

Grids

Outline Principles Subgrid denition Convolution function MPI communications Applying NICAS Conclusions 9

Convolution

MPI

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

Choice of the convolution function Gaspari and Cohn (1999) function of compact support radius r → homogeneous normalized distance

10

dij0 =

dij r

Conclusions

Principles

Grids

Convolution

MPI

Application

Choice of the convolution function Gaspari and Cohn (1999) function of compact support radius r → homogeneous normalized distance

10

dij0 =

dij r

Conclusions

Principles

Grids

Convolution

MPI

Application

Choice of the convolution function Gaspari and Cohn (1999) function of compact support radius r → heterogeneous normalized distance

10

dij0 = q

dij

(ri2 + rj2 )/2

Conclusions

Principles

Grids

Convolution

MPI

Application

Conclusions

Homogeneous / heterogenous support radius Homogeneous support radius → homogenous subgrid:

11

Principles

Grids

Convolution

MPI

Application

Conclusions

Homogeneous / heterogenous support radius Heterogenous support radius → heterogenous subgrid:

11

Principles

Grids

Convolution

MPI

Application

Conclusions

Homogeneous / heterogenous support radius Convolution with an homogenous support radius

11

Principles

Grids

Convolution

MPI

Application

Conclusions

Homogeneous / heterogenous support radius Convolution with an heterogeneous support radius

11

Principles

Grids

Convolution

MPI

Application

Sharp support radius gradients Gaspari and Cohn (1999) function of compact support radius r → heterogeneous normalized distance

12

dij0 = q

dij

(ri2 + rj2 )/2

Conclusions

Principles

Grids

Convolution

MPI

Application

Sharp support radius gradients Gaspari and Cohn (1999) function of compact support radius r → heterogeneous normalized distance

12

dij0 = q

dij

(ri2 + rj2 )/2

Conclusions

Principles

Grids

Convolution

MPI

Application

Sharp support radius gradients Gaspari and Cohn (1999) function of compact support radius r → heterogeneous normalized distance

12

dij0 = q

dij

(ri2 + rj2 )/2

Conclusions

Principles

Grids

Convolution

MPI

Application

Conclusions

Sharp support radius gradients Gaspari and Cohn (1999) function of compact support radius r → heterogeneous normalized distance

12

j −1

dij0 = ∑ dk0 ,k +1 (network) k =i

Principles

Grids

Convolution

MPI

Application

Sharp support radius gradients

Convolution function for a homogeneous support radius r 12

Conclusions

Principles

Grids

Convolution

MPI

Application

Sharp support radius gradients

Heterogeneous support radius r and subset S2c (black dots) 12

Conclusions

Principles

Grids

Convolution

MPI

Application

Conclusions

Sharp support radius gradients

12

Convolution function for a heterogeneous support radius r with the distance-based approach; zone of lower r (dotted)

Principles

Grids

Convolution

MPI

Application

Conclusions

Sharp support radius gradients

12

Convolution function for a heterogeneous support radius r with the network-based approach; zone of lower r (dotted)

Principles

Grids

Convolution

MPI

Application

Conclusions

Complex boundaries

Convolution functions with complex boundaries of the NEMO grid: • for a distance-based approach (left) • for a network-based approach (right) 13

Principles

Grids

Convolution

MPI

Application

Conclusions

Subgrid resolution For a given support radius, a resolution parameter ρ denes the subgrid density:

14

ρ = 8 (2827 points)

Principles

Grids

Convolution

MPI

Application

Conclusions

Subgrid resolution For a given support radius, a resolution parameter ρ denes the subgrid density:

14

ρ = 6 (1590 points)

Principles

Grids

Convolution

MPI

Application

Conclusions

Subgrid resolution For a given support radius, a resolution parameter ρ denes the subgrid density:

14

ρ = 4 (706 points)

Principles

Grids

Convolution

MPI

Application

Conclusions

Subgrid resolution For a given support radius, a resolution parameter ρ denes the subgrid density:

14

Horizontal convolution function for a spectral method and for the NICAS method with a decreasing resolution (ρ = 8, 6 and 4)

Principles

Grids

Convolution

MPI

Application

Conclusions

Subgrid resolution For a given support radius, a resolution parameter ρ denes the subgrid density:

14

Vertical convolution function for a spectral method and for the NICAS method with a decreasing resolution (ρ = 8, 6 and 4)

Principles

Grids

Convolution

MPI

Application

Conclusions

Normalization computation Prohibitive cost of applying the non-normalized method to a Dirac at every gridpoint (O (n2 )). However, very limited number of nodes involved in the computation of Nii . Number of non-zero nodes: • 1 node of G f in δ , i • up to 3 nodes of G h in ShT δ (bilinear interp.) i • up to 6 nodes of G v in Sv T ShT δ (linear interp.) i • up to 18 nodes of G s in Ss T Sv T ShT δ (bilinear interp.) i Aordable procedure: 1. computing the 18 non-zero values of Ss T Sv T ShT δi (→ δ 0 ) 2. applying the relevant block of the subgrid convolution matrix Cs to these coecients (→ δ00 ) 3. computing the normalization coecient as the inverse

 square-root of a scalar product (Nii = δ 0 , δ 00 −1/2 ) 15

Principles

Grids

Outline Principles Subgrid denition Convolution function MPI communications Applying NICAS Conclusions 16

Convolution

MPI

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

MPI communications Four dierent halo zones: • [F] zone: full grid G f nodes on a given task (usually provided by the model itself) • [A] zone: subgrid G s nodes on a given task (dene by the subsampling on [F]) • [B] zone: subgrid G s nodes involved in the interpolation • [C] zone: subgrid G s nodes involved in the convolution

Ce = 17

[F]

N [F] S [B]←≺[A]←≺[C] Cs [C]←[B] ST [F] NT [F]

Conclusions

Principles

Grids

Convolution

MPI

Application

MPI communications Four dierent halo zones: • [F] zone: full grid G f nodes on a given task (usually provided by the model itself) • [A] zone: subgrid G s nodes on a given task (dene by the subsampling on [F]) • [B] zone: subgrid G s nodes involved in the interpolation • [C] zone: subgrid G s nodes involved in the convolution

Ce = 17

[F]

N [F] S [B]←≺[A]←≺[C] Cs [C]←[A]←≺[B] ST [F] NT [F]

Conclusions

Principles

Grids

Outline Principles Subgrid denition Convolution function MPI communications Applying NICAS Conclusions 18

Convolution

MPI

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

Conclusions

Namelist Model grid

NICAS

Oine computations Support radii (if heterogeneous)

Model grid splitting

Compute parameters and normalization

Compute MPI distribution

All parameters

art

t res

res Parameters for proc 1 Parameters for proc 2 Parameters for proc p

19

t tar

Optional tests (e.g. adjointness, dirac application)

Dirac test results

Principles

Grids

Convolution

MPI

Application

Conclusions

Inline computations Generic implementation pattern for all linear operations (horizontal and vertical interpolations, convolution): fld_out = 0.0 do i_s=1,n_s fld_out(row(i_s)) = fld_out(row(i_s)) + S(i_s)*fld_in(col(i_s)) end do

where fld_in is the input eld, fld_out the output eld, n_s the number of operations involving row and col as indices and S as coecients. Easy to code the adjoint operator:

20

fld_out = 0.0 do i_s=1,n_s fld_out(col(i_s)) = fld_out(col(i_s)) + S(i_s)*fld_in(row(i_s)) end do

Principles

Grids

Convolution

MPI

Generic implementation Subset S2c for a heterogeneous support radius r, for the regional model AROME:

21

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

Generic implementation Convolution functions for a heterogeneous support radius r, for the regional model AROME:

21

Conclusions

Principles

Grids

Convolution

MPI

Application

Conclusions

Scalability Elapsed time for one application of the correlation with ARPEGE on a TL149 grid:

22

Principles

Grids

Convolution

MPI

Application

Conclusions

Scalability Elapsed time for one application of the correlation with ARPEGE on a TL399 grid:

22

Principles

Grids

Outline Principles Subgrid denition Convolution function MPI communications Applying NICAS Conclusions 23

Convolution

MPI

Application

Conclusions

Principles

Grids

Convolution

MPI

Application

Conclusions

Conclusions Already coded : • Generic Fortran/C++ code for the NICAS method (oine computations): • using ESMF for horizontal interpolation • convolution with a Gaspari and Cohn (1999) function, possibly • • • •

•

heterogeneous and with complex boundaries exact normalization approximate square-root formulation hybrid MPI/OpenMP parallelization basic tests (adjoint, positive-deniteness, Diracs)

Inline computations eciency tested in ARPEGE and AROME models → satisfying scalability

To be done: • Further testing, validation, documentation • Inline implementation in a exible framework... 24