Outline
Uniform and non-uniform pseudo-random numbers generators for high dimensional applications Régis LEBRUN EADS Innovation Works
Formation Échantillonnage Mercredi 9 mai 2012
©EADS IW 2012
Formation Échantillonnage
Outline
Outline
1
Pseudo-random numbers generator (PRNG)
2
Uniform PRNGs
3
Non-uniform PRNGs
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Pseudo-random numbers generator (PRNG)
Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin. – John Von NEUMANN (1951) cited in The Art of Computer Programming, Donald E. KNUTH
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Stream of independent, U(0, 1) random variables
Uniformity and independence The basic building block of stochastic simulation software is the uniform random numbers generator. We are looking for a routine capable of producing independent, identically distributed realizations of the uniform distribution over [0, 1]. We want to do it algorithmically and deterministically !
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
What is a pseudo-random number generator ? Definition, from [1] A pseudo-random number generator is a structure G = (S, s0 , T , U, G ) where : S is a finite set of states, s0 ∈ S is the initial state (or seed), the mapping T : S → S is the transition function, U is a finite set of output symbols, and G : S → U is the output function. The state of the generator evolves according to the recurrence sn = T (sn−1 ) and the generator outputs the number un = G (sn ), called the pseudo-random numbers.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
What is a pseudo-random number generator ? Some comments Nothing about randomness in this definition ! A PRNG is a dynamic system partially observed. G is not always bijective. Depending on the type of U (double, float), cardU cardS ! Will be periodic, with a period P ≤ cardS. Such a device is suppose to be able to produce a stream of numbers that "looks like" a sequence of realizations of iid uniform random variables, i.e. that is not statistically different from a true random sequence of random numbers. The attempt to define what is a random sequence fills more than 30 pages in [2]... Uniformity over [0, 1] cannot be achieved on a computer due to the finite precision of numbers representation, but discrete uniform distribution over Ω = {0/m, 1/m, . . . , (m − 1)/m} can be (quite well) approximated through the equirepartition property over Ω, where m = cardU.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
What is a pseudo-random number generator ? Two delicate points Computers are able to perform exact integer arithmetic over Z/qZ and implements algorithms that approximate uniform distribution on integers {0, . . . , q − 1}. For example, in most implementations : S = (Z/pZ)k where (p = 2, k = 104 ) (high bits buckets) or (p = 232 , k = 1) (for congruential generators) U = Z/qZ whith q the highest possible, such as the integers {0, . . . , q − 1} are easily represented with the computer architecture (32 or 64 bits) and G easy to implement (for congruential generators, q = p). But probability and statistics algorithms need a uniform distributino over [0, 1] ! So two delicate points : the choice of the strategy to transform integers of U into reals of [0, 1]. This strategy is chosen by the developper of the statistical software. the precision of the representation of reals over [0, 1]
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
What is a pseudo-random number generator ?
Which strategy to transform integers into reals of [0, 1] ? This strategy is chosen for you by the developper of the statistical software. For U = Z/qZ, we define the real x ∈ [0, 1] by : u Statregy S1 : x = q „ « u Statregy S2 : x = 1 + −1 q Which difference ? ...
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Which strategy to transform integers into reals of [0, 1] ? Some basics on informatic types : floatting point numbers The computer compilator implements 2 types of reals from the memory words of the hardware : simple precision (or float or single) : reals encoded with 32 bits. The first bit is the sign ± ; the 8 following encodes the exposant (power of 2) ; the 23 others are the mantissa. double precision (or double) : reals encoded with 64 bits. The first bit is the sign ± ; the 11 following encodes the exposant (power of 2) ; the 52 others are the mantissa. Float
+/−
Double
+/−
8 : exposant
23 : mantissa
11 : exposant
52 : mantissa
Remark : A memory word 32 bits (resp. 64 bits) is easily interpreted as a int32, uint32 and float (resp. int64, uint64 and double) : they are called types 32 bits (resp. types 64 bits). ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Which strategy to transform integers into reals of [0, 1] ? Repartition of the floating points over the real line The repartition of floating points on the real line is not homogeneous. 23
2 reals
23
23
2 reals
2 reals
Float 0
1/8 1/4
1/2
1
part 2
2 part 1
The exposant fixes the interval [2q , 2q+1 [ and the mantissa generates the 223 reals within the interval. 52
2 reals
52
52
2 reals
2 reals
Double 0
1/8 1/4
1/2 part 2
1
2 part 1
The exposant fixes the interval [2−(q+1) , 2−q [ and the mantissa generates the 252 reals within the interval.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Which strategy to transform integers into reals of [0, 1] ? Quantization error „ « u The sequence has to be inserted in the floating points irregular grid q u=0,...,q−1 on [0, 1] : each real is rounded to the nearest floating point in this grid. It generates the quantization ε1 . „ error « u The sequence 1 + has to be inserted in the floating points regular q u=0,...,q−1 grid on [1, 2]. It generates the quantization error ε2 . The translation by −1 generates no additional quantization error since the translated grid is a sub grid of the grid over [0, 1].
u/q
1+u/q
floating points 0
1/8 1/4
1/2 part 2
©EADS IW 2012
1
2 part 1
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Which strategy to transform integers into reals of [0, 1] ? What is the difference between S1 and S2 ? The number of floating points inside intervals of type [2−(k+1) , 2−k ] is given by 2m (mantissa). Imagine q 2m then : each floating point of [1, 2] or [0, 1] are selected at most once. In that case, ε1 = ε2 . not selected floating point
0
1/8 1/4
1/2
1
2
part 2
part 1
0.8 0.6 0.4 0.2
0.0
loi S1/S2 loi continue
0.0
0.2
0.4
0.6
0.8
1.0
loi discrete loi continue
0.0
0.2
0.4
PDF
CDF
0.6
0.8
1.0
CDF Loi continue / Loi discrete
1.0
Loi discrete S1 / S2
0.0
0.2
x
0.4
0.6
0.8
Regular grid over [0, 1] Homogeneous resolution of [0, 1] S1 = S2
1.0
u
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Which strategy to transform integers into reals of [0, 1] ? What is the difference between S1 and S2 ? Imagine q 2m then : all the floating points of [1, 2] are selected several times, those of [1/2, 1] have all been selected once, those of [1/4, 1/2] have been selected at most once, and so on. In that case, ε1 6= ε2 : saturated zone not selected floating point
0
1/8 1/4
1/2
1
2
part 2
part 1
1.0
Loi discrete S1 / S2
0.20
Loi discrete S1 / S2
CDF 0.4
0.10
0.2
0.05
loi continue loi discrete loi S1
0.0
0.00
PDF
0.6
0.15
0.8
loi S1 loi S2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
x
0.4
0.6
0.8
Irregular grid over [0, 1] Non Homogeneous resolution of [0, 1] S1 6= S2
1.0
x
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Which strategy to transform integers into reals of [0, 1] ? Which impact when S1 6= S2 ? Strategy S1 : the grid over [0, 1] is irregular , we can note : drawbacks : 1 P(0 < U < x) 6= P(1 − x < U < 1) for small x, 2 different resolution around x = 0 and x = 1 leads to different estimation of extreme quantiles : lowest quantiles are more precisely estimated than highest ones. 3 ... and for any distribution generated by the CDF inversion method from the uniform distribution ! advantages : the evaluation of P(0 < U < x) for small x is accurate. Strategy S2 : the grid over [0, 1] is regular , we can note : inconvenient : P(0 < U < x) or P(1 − x < U < 1) for small x is not very accurate, advantages : 1 The continuous equality P(0 < U < x) = P(1 − x < U < 1), ∀x is recovered at the discrete level, 2 same accuracy for the lowest and highest quantiles. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Which strategy to transform integers into reals of [0, 1] ? Softwares choice In R : U = Z/232 Z, (q = 232 ) reals are double (m = 52) =⇒ case where q 2m : S1 = S2 In RandLib (Fortran library used in Scilab) : U = Z/232 Z, (q = 232 ) reals are single (m = 23) =⇒ case where q 2m : S1 6= S2 =⇒ Strategy chosen : S1 : precision 2−31 close to 0 and 2−23 close to 1. (if S2 , precision 2−23 everywhere). In Open TURNS : U = Z/264 Z, , (q = 264 ) reals are double (m = 52) =⇒ case where q 2m : S1 6= S2 =⇒ Strategy chosen : S2 : precision 2−52 everywhere. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Precision of the representation of reals
Uniformity : Impact of the type of U Imagine strategy S2 . If U is a float ∈ [0, 1] : m = cardU = 223 then there exists one number < 10−14 : 0 : P(U < 10−14 ) ' P(Um < 10−14 ) = P(Um = 0) =
1 ' 10−8 223
If U is a double ∈ [0, 1] : m = cardU = 252 then there exists 45 numbers < 10−14 : P(U < 10−14 ) ' P(Um < 10−14 ) =
k=45 X k=1
P(Um = uk ) =
45 ' 9.910−15 252
Conclusion : For rare events, the type of the numbers produced by the PRNG has an important impact !
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
What is a good pseudo-random number generator ? Uniformity Uniformity over [0, 1] cannot be achieved on a computer due to the finite precision of numbers representation, but discrete uniform distribution over Ω = {0/m, 1/m, . . . , (m − 1)/m} can be (quite well) approximated through the equirepartition property over Ω, where m = cardU.
Remark : The precision of the approximation depends on the type of the output symbols U ! ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
What is a good pseudo-random number generator ?
Independence ... even if sn = T (sn+1 ) ! Independence between realizations cannot be achieved due to the deterministic nature of the recurrence in the generator, but can be (quite well) approximated due to the mixing property of T . This last point may be made more consistent using the copula theory. A possible way to prove it : suppose that s0 is a random variable uniformly distributed over [0, 1], check that the copula of (s0 , T (s0 ), . . . , T n (s0 )) is a shuffle of min and check that it is close to the independent copula. There is a hope to succeed in this task, thanks to Mikusiński’s theorem.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Empirical validation of PRNGs Statistical and Analytical tests So, a PRNG has to be extensively checked to be used in serious simulations. The tests must check : The uniformity of the PRNG output : Kolmogorov-Smirnov, Chi-square, gap tests. The independence of the PRNG output : serial test, spectral test, autocorrelation These tests can be classified into : Empirical tests : one tests for a property on a large sample produced by the PRNG Analytical tests : one checks a property on the whole set U of possible output of a PRNG Linear Congruential Generators We will illustrate the potential pitfalls in designing a PRNG through the example of Linear Congruential Generators and more precisely the Multiplicative Congruential Generators.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Linear congruential generators (LCG) Definition Let m > 0 (the modulus), a > 0 (the multiplier) and c (the additive constant) be integers. The linear congruential generator associated with the triplet (m, a, c) is given by : S = Z/mZ = {0, . . . , m − 1} for c > 0 and S = {1, . . . , m − 1} for c =0 sn = T (sn−1 ) = a.sn−1 + c mod m U = {0, 1/m, . . . , 1 − 1/m} for c > 0 and U = {1/m, . . . , 1 − 1/m} for c = 0 un = G (sn ) = sn /m When c = 0, these generators are called multiplicative congruencial generators. They are often used due to the extra gain in speed. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Linear congruential generators Properties It is the first class of PRNGs for which a mathematical theory has been built, They are fast, They are easy to implement, They are (still) widespreads (drand()∗ , drand48()∗ ...) But they have serious weaknesses : Short period in typical implementations (m = 232 ' 5.109 ) ; Bad repartition in high dimension ; Lower order bits have a shorter period than full numbers (so don’t use yn = sn mod M if you want integers in {0, . . . , M − 1} from integers in {0, . . . , m − 1} with M ≤ m). (∗) : drand() is a LCG with m = 232 : S are integers encoded with 32 bits. (∗) : drand48() is a LCG with m = 248 : S are integers encoded with 48 bits. It aims to produce floats (32 bits) and to produce doubles, it fills the lowest bits with 0. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators (MCG) How to choose m, a ? If m = 2M , M ≥ 3, the maximal period is 2M−2 , attained for all odd seed if a mod 8 ≡ 3 or 5. The period m − 1 is attained only if m is prime. In this case, the period divides m − 1, and is m − 1 if and only if a is a primitive root of m − 1, that is, a 6= 0 and a(m−1)/p 6≡ 1 mod m for each prime factor p of m − 1. More to read in [2]. Problem with high dimension Suppose that one want to sample uniformly the hypercube [0, 1]d . The natural way to do this is to consider the points (udn , udn+1 , ..., udn+d −1 )n≥0 .
u1 u2 u3
um
One can generate at most m (or m − 1) points in [0, 1]d , which will necessarily be very scarcely distributed in high dimension. In the case of LCGs or MCGs, these points appear to be distributed according to a very regular pattern : a lattice. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators Problem with high dimension
Lattice obtained with random() (standard C library) in dimension 2. Equirepartition : ok .... but satisfactory ? What about independence ?
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
Bad high dimension properties : Marsaglia’s theorem, from [3] Let d > 0 be a positive integer. If α0 , . . . , αd−1 is any choice of integers such that : α0 + · · · + αd−1 ad−1 ≡ 0 mod m (1) then all points of the form (un , . . . , un+d−1 ) will lie in the set of parallel hyperplanes defined by the equations : α0 x0 + · · · + αd−1 xd−1 = 0, ±1, ±2, . . .
(2)
There are at most |α0 | + · · · + |αd−1 | of these hyperplanes which intersect ]0, 1[d , and there is always a choice of α0 , . . . , αd−1 such that all of the points fall in fewer than (d !m)1/d hyperplanes.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
Bad high dimension properties These values are given for the best MCG... ( it means for an optimized choice of (a, m)). d =3 d =4 d = 5 d = 6 d = 7 d = 8 d = 9 d = 10 m = 223
369
119
63
42
32
27
24
22
232
2953
566
220
120
80
60
48
41
m = 252
300079
18131
3520
1216
582
340
227
166
m = 264 4801279 145055
18578
4866
1910
963
573
382
m=
223 for float, 232 for uint32, 252 for double, 264 for uint64
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators Bad high dimension properties No point falls between two hyperplanes. At least one of these slices has a volume V 1 where N is the number of hyperplanes. such as V > N Example 1 : for m = 232 (float precision) and d = 3, there exists one slice of 1 probability = 3.10−4 which is never explored by the MCG. 2953 Risks : If you analyse an event which probability is inferior to 3.10−4 , it might be included in this unexplored space ! Example 2 : for m = 252 (double precision) and d = 10, there exists one slice of 1 probability = 6.10−3 which is never explored by the MCG. 166 Risks : If you analyse an event which probability is inferior to 6.10−3 , it might be included in this unexplored space !
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators
Bad high dimension properties Weights of some zones unexpored by the MCG random generators : d =3 d =4 d =5 d =6 d =7 d =8 d =9
d = 10
m = 223
2.7e-3
8.4e-3
1.6e-2
2.4e-2
3.1e-2
3.7e-2
232
3.4e-4
1.8e-3
4.5e-3
8.3e-3
1.2e-2
1.7e-2
2.1e-2
2.4e-2
m = 252
3.3e-6
5.5e-5
2.8e-4
8.2e-4
1.7e-3
340
2.9e-3
6.0e-3
m = 264
2.1e-8
6.9e-6
5.4e-5
2.0e-4
5.2e-4
1.0e-3
1.7e-3
2.6e-3
m=
©EADS IW 2012
Formation Échantillonnage
4.2e-2
4.5e-2
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Analytical tests The gap test This test is one of the most basic ones. It aims at testing if a PRNG associated with a measure µ produces ouputs U that cover evenly the support of µ. More precisely : Let ρ = 1/cardU 1 be the theoretical optimal resolution of a PRNG ; Find µ(I ∗ ) =
max
I =]si ,sj [,I ∩U=∅ µ(I ∗ ) = O(ρ) :
µ(I ), where si , sj ∈ U ;
Check that the PRNG does not leave significant gaps in the support of µ. One can note that a LCG with full period is perfect with regard to this test, as it has a gap value of ρ, or ρ/(1 − ρ) for MCGs. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Analytical tests The spectral test The spectral test is specifically designed to detect non-uniformity in the multidimensional repartition of U in [0, 1]d . It was designed after a period of massive use of LCGs that where very poorly designed (as the famous RANDU shipped with IBM computers in the 80’). The test is defined by : Compute the quantity : (
) d−1 X d i νd = min ||α||2 α ∈ Z , αi a ≡ 0 mod m
(3)
i=0
for d = 2, . . . , dmax . The quantity (νd )−1 is related to the largest gap between adjacent hyperplanes in a family of parallel hyperplanes that covers the d − dimensional points generated by the LCG.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Analytical tests The spectral test X2 d
α
1/α2 1/α1
X1
Compute the normalized quantity : µd =
π d/2 νdd Γ(d /2 + 1)m
(4)
The test is successful if µd > 0.1 for d = 2, . . . , 6. This test has been experimentaly proved to be very effective in separating the good PRNGs from the bad ones. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Other related PRNGs
LCGs and derived not better than MCGs Marsaglia’s theorem can be extended to more general LCGs : research of a family of hyperplanes that covers the points of a sample. Even if the details change, the basic fact remains : LCGs are bad for high dimensional applications unless one is ready to use a HUGE value for m (m > 101500 ), for which one can work in dimension d = 100 without too much care (ie N = cardU : not possible to regroup the points in fewer hyperplanes). Generators that chain several LCGs suffer from the same defect : it is just a way to adjust m to higher values, but starting from m = 232 one cannot attain sufficient resolution without an obscene number of steps. Indeed, the combination of k LCGs with moduli m1 , . . . , mk has a maximum period of (m −1)...(m −1) P = 1 k−1 k ' 232(k−1) . 2
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
What about Excel and Crystal Ball ? Excel and Add-ins Excel In Excel XP, cardS = 216 = 65 536 so the period is < 216 and has effect on samples of size > 256 ... In Excel 2003 and Excel 2007, the PNRG has been enhanced and is a congruential multiple generator (combination of 3 multiplicative congruential generator) that proves to have a period of 2 1013 which enables to generate about 107 numbers without period effect in 1D studies. Add-ins provide some algorithms which replace those of Excel. Most of them use the Mersenne Twister algorithm. Crystal Ball (Oracle) Crystal Ball : This is a well-known Excel plugin for uncertainty quantification and propagation published by Oracle. The PRNG is a multiplicative congruential generator which formula is Rn+1 = (62089911Rn ) mod(231 − 1). It remains the same for all the versions of the plugin up to the current revision (as of 04-20-2012). Its period is T = 231 − 2 √ =⇒ be carefull to samples of size more than 231 − 2 = 46340 ! Furthermore no equirepartition nor uniformity on [0, 1]n for n ≥ 3. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Better than LCGs
SIMD-oriented Fast Mersenne Twister (SFMT), from [5] Three of the most important properties of a PRNG are : a huge period ; good equirepartition properties even in high dimension ; fast generation of random numbers. The SIMD-oriented Fast Mersenne Twister fulfills these needs with a period of 219937 − 1 ' 106001 (can be easily extended to 2216091 − 1 ' 1065050 ), a provable equirepartition in dimension up to 382 (resp. 4077) as a double-precision floating-point uniform random numbers generator.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
SFMT details An efficient Linear Feedback Shift Register generator The state space S of this generator is [(Z/2Z)w ]N : an element s in S is seen as an N dimensional vector (x0 , . . . , xN−1 ) over (Z/2Z)w . Each xk is called a register. The transition function T is linear and very sparse and can be seen as an N terms linear recurrence over (Z/2Z)w and takes the form : T : ((xn )0 , . . . , (xn )N−1 ) = ((xn−1 )1 , . . . , (xn−1 )N−1 , z) with z = g (((xn−1 )0 , . . . , (xn−1 )N−1 ))
(5)
with g a sparse linear function. For SFMT, w = 128 in order to store the xk in SIMD (Single Instruction, Multiple Data) registers of modern CPUs, and the function g is such that : g (x0 , . . . , xN−1 ) = x0 A + xM B + xN−2 C + xN−1 D ©EADS IW 2012
Formation Échantillonnage
(6)
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
SFMT details An efficient Linear Feedback Shift Register generator The parameters M, N, A, B, C and D have been chosen in order to have a period multiple of 219937 − 1 and provide equirepartition in high dimension, which leads to, for x ∈ (Z/2Z)128 : N = 156 and M = 122, 128
xA = (x > 11) ⊕ (BFFFFFF 6 BFFAFFFF DDFECB7F DFFFFFEF ), 128
xC = (x >> 8), 32
xD = (x 0 such that : ∀x ∈ supp(f ), h(x) ≤ f (x) ≤ λg (x) Then, the algorithm is :
g
λg
1
Generate Y according to g ;
2
Generate U uniformly over [0, λg (Y )] ;
3
If U < h(Y ), accept Y as a realization of f ;
4
If U < f (Y ), accept Y as a realization of f ;
5
else go back to step 1 ;
h f y
©EADS IW 2012
Formation Échantillonnage
(7)
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Rejection with squeeze Properties This algorithm uses at least two realizations of the underlying uniform PRNG, and its efficiency is measured in term of the mean number of iterations needed to produce one non-uniform realization. The squeeze function is here only to postpone the potentially expansive test in step 4. The attainable range is extended compared to the CDF inversion method, as one have to compare functions that takes values close to 0 instead of 1. For the normal distribution generated using exponential tails, one can attain the range [−13.47, 13.47] for floats and [−37.62, 37.62] for doubles for a standard normal distribution. The gap test can be severely degraded if the rejection distribution is far from f : this method change the frequency of the realizations of g , not their values, so if they are not well distributed with respect to f it can lead to significant gaps. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Transformation method
Principle A random variable X one want to sample can sometimes be expressed as a function of one or several other independent random variables ξ1 , . . . , ξN not necessarily identically distributed, but that can all be easily sampled : X = φ(ξ1 , . . . , ξN )
(8)
The simulation method is then straightforward : sample each of the ξi , then compute a realization of X using φ.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Transformation method Properties Many possibilities to express X as a function of ξi ; Straightforward implementation once the decomposition has been found ; Can be expansive in term of calls to the uniform PRNG for one realization of X , even if it is sometimes possible to get several iid realizations of X for one realization of the ξi , using differente functions. The effective range of the resulting PRNG can be limited. For example, for the normal distribution, the Box-Muller transformation leads to the range [−5.64, 5.64] for floats and [−8.49, 8.49] for doubles for a standard normal distribution. The gap test can be severely degraded if the underlying uniform PRNG has a specific structure that interact with the structure of φ. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
LCGs and the ratio of uniforms method The ratio of uniforms method This method is a mix between the transformation method and the rejection method, that leads to fast and generic non-uniform PRNGs. p Let (U, V ) be uniformly distributed over C = {(u, v ) | 0 ≤ u ≤ c.f (v /u)}, then X = V /U has density f . If f has sub-quadratic tails, the area of C is bounded and C can be enclosed in a rectangle [0, u + ] × [v − , v + ]. It is then possible to sample (U, V ) by the rejection method, and it is very often possible to derive efficient squeeze functions.
v v+
u+ u
v− ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
LCGs and the ratio of uniforms method Gap default when combining LCGs and the ratio of uniforms method, from [7] If (U, V ) is obtain using consecutive output of a LCG(m, a, c), as these points are contained within a lattice, the possible values for the ratio V /U will inherit a very special structure. It is then possible to show that : [−v + , v + ], then there exist a gap g such Th. 1 If C = [0, u + ] × q that µ(g ) ≥
√1 m
√
3 32
Th. 2 If C = [0, u + ] × q [v − , v + ], then there exist a gap g such √
2 that µ(g ) ≥ √1m 163 q2q+1 where q2 = |e1 |/|e2 | is the 2 Beyer quotient of the lattice, (e1 , e2 ) being the Minkowski-reduced basis of the lattice.
These lower bounds are much worse than the expected O(1/cardU) = O(1/m) ! ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
LCGs and the ratio of uniforms method Gap for usual distributions We consider the Cauchy, normal and exponential distributions for LCGs with m ' 232 (gap = 1/m ' 2.3 10−10 on the realisations of u and also for the distribution) and q2 = q2 (m, a) varying from 0.1 to 1 (rather good LCGs). We compute the lower bound of µ(g ) scaled by 106 : q2 0.1 0.5 1.0
Cauchy 4.52 4.52 4.52
Normal 2.08 4.17 4.65
Exponential 2.98 5.58 6.12
If you are dealing with (not so) rare events, don’t use the combination LCG/ratio of uniforms !
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Fast and high-quality normal PRNG
The ziggurat method (Marsaglia, Doornik, see [8]) The ziggurat method is an innovative use of the rejection method with squeeze. Its key characteristics are : allow to sample symmetric unimodale continuous random variables or positive continuous random variables with decreasing PDF. generate fast and high quality random deviates The main idea is to use rejection with squeeze with respect to horizontal histograms, with a specific treatment of the tail of the distribution. We detail this method in the case of the standard normal distribution.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Fast and high-quality normal PRNG The ziggurat method (Marsaglia) How to build the ziggurat ? Restrict the PDF f to x ≥ 0 ; Cover its graph with horizontal rectangular bands The bottom band contains also the tail of PDF ; All these bands have the same area V ; Number these bands from 1 (top) to N (bottom) ; Consider the inner horizontal histogram as an inner squeeze function.
©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Fast and high-quality normal PRNG The ziggurat method (Marsaglia) How to use it ? 1
Generate a band number i uniformly distributed over {1, . . . , N} ;
2
If i < N generate ui uniformly in [0, xi ] where xi is the length of the covering band, else generate ui uniformly in [0, xN = V /f (xN−1 )].
3
If ui ≤ xi −1 (with x0 = 0), accept ui
4
Else 1
2
if i = N, generate ui in the tail e.g. using Marsaglia’s method ; else use simple rejection in the wedge. ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Fast and high-quality normal PRNG Generating realizations from the tail of the normal distribution, Marsaglia’s method To generate a random variate X with density proportional to exp(−x 2 /2)1{x≥a} , do : 1
Generate U1 and U2 , two iid random variables uniform on ]0, 1] ;
2
Define X =
3
If x 2 < −2y return a − X . Note : X < 0
1 a
log(U1 ) and Y = log(U2 ) ;
The joint density of the pairs (X , Y ) obtained after a successful step 3 has a joint density of the form : pXY (x, y ) =
K exp(x/a)1{x 0} ; ©EADS IW 2012
Formation Échantillonnage
Pseudo-random number generators Uniform PRNGs Non-uniform PRNGs
Multiplicative congruential generators Demonstration (3/3) 4
There exist a set α0 , . . . , αd−1 not all zeros with |α0 | + · · · + |αd−1 | ≤ (d !m)1/d because (1) is equivalent to show that there exist integers t0 , . . . , td−1 not all zeros such that : |mt0 − at1 | + |t1 − at2 | + · · · + |td−2 − atd−1 | + |td−1 | ≤ (d !m)1/d by writing that
0 ··· ··· 0 .. .. −a . 1 . (α0 , . . . , αd−1 ) = (t0 , . . . , td−1 ) 0 . . . . . . . . . ... . . .. ... ... 0 .. 0 ··· 0 −a 1 and it is true as stated in the preliminary theorem (the determinant of the matrix is m). ©EADS IW 2012
m
Formation Échantillonnage
