Black-Box Optimization Benchmarking Template for the Comparison of

Jan 8, 2015 - 15/15. Table 24: ERT on /24 in 5-D over ERTbest obtained in BBOB 2009. ∆fopt 1e1. 1e0. 1e-1. 1e-3. 1e-5. 1e-7. #succ f24 1622. 2.2e5. 6.4e6.
Télécharger le PDF
849KB taille 6 téléchargements 309 vues
commentaire
Report
Black-Box Optimization Benchmarking Template for the Comparison of Algorithms on the Noiseless Testbed January 8, 2015

1

Results

Results from experiments according to [?] on the benchmark functions given in [?, ?] are presented in Figures ??, ?? and ??. The expected running time (ERT), used in the figures and table, depends on a given target function value, ft = fopt + ∆f , and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach ft , summed over all trials and divided by the number of trials that actually reached ft [?, ?]. Statistical significance is tested with the rank-sum test for a given target ∆ft using, for each trial, either the number of needed function evaluations to reach ∆ft (inverted and multiplied by −1), or, if the target was not reached, the best ∆f -value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial under consideration if available. Tables ?? to ?? and ?? to ?? give the Expected Running Time (ERT) for targets 101, −1, −3, −5, −7 divided by the best ERT obtained during BBOB-2009 (given in the ERTbest row), respectively in 5-D and 20-D. Bold entries correspond to the best (or 3-best if there are more than 3 algorithms) values. The median number of conducted function evaluations is additionally given in italics, if ERT(10−7 ) = ∞. #succ is the number of trials that reached the final target fopt + 10−8 . Entries with the ↓ symbol are statistically significantly better (according to the rank-sum test) compared to the best algorithm in BBOB-2009, with p = 0.05 or p = 10−k where k > 1 is the number following the ↓ symbol, with Bonferroni correction of 24.

1

6

15 Rastrigin

5 4 3 2 1 0 ftarget=1e-08 2 3 5 6

17 Schaffer F7, condition 10

6

5

5

4

4

3

3

2

2

1

1

0 ftarget=1e-08 2 3 5

10

20

21 Gallagher 101 peaks 7 6 5 4 3 2 1 0 ftarget=1e-08 2 3 5 10 20

18 Schaffer F7, condition 1000

0 ftarget=1e-08 40 2 3 5

10

20

22 Gallagher 21 peaks 7 6 5 4 3 2 1 0 ftarget=1e-08 40 2 3 5 10 20

10

20

16 Weierstrass 7 6 5 4 3 2 1 0 ftarget=1e-08 40 2 3 5 10 20

7 19 Griewank-Rosenbrock F8F2 6 5 4 3 2 1 0 ftarget=1e-08 40 2 3 5 10 20 40

23 Katsuuras 7 6 5 4 3 2 1 0 ftarget=1e-08 40 2 3 5 10 20

6

40

20 Schwefel x*sin(x)

5 4 3 2 1 0 ftarget=1e-08 2 3 5

10

20

24 Lunacek bi-Rastrigin 7 6 5 4 3 2 abipop_bbob abipop_bbob_history 1 abipop_bbob_history_best 0 ftarget=1e-08 40 2 3 5 10 20

40

40

Figure 1: Expected running time (ERT in number of f -evaluations) divided by dimension for target function value 10−8 as log10 values versus dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24 . Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Horizontal lines give linear scaling, slanted dotted lines give quadratic scaling. Black stars indicate statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six). Legend: ◦:abipop bbob, O:abipop bbob history, ?:abipop bbob history best.

2

separable fcts

moderate fcts

Proportion of function+target pairs

ill-conditioned fcts

multi-modal fcts 1.0 f15-19,5-D 0.8

abipop abipop_bbob_history_best

0.6 0.4

abipop abipop_bbob_history

0.2 0.00

1

abipop_bbob abipop

2 3 4 5 6 7 8 log10 of (# f-evals / dimension) all functions

abipop abipop_bbob_history

0.6 0.4

best 2009 best 2009

0.2 0.00

Proportion of function+target pairs

Proportion of function+target pairs

weakly structured multi-modal fcts 1.0 f20-24,5-D 1.0 f15-24,5-D abipop_bbob_history_best abipop 0.8

1

abipop_bbob abipop

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

best 2009 best 2009

abipop_bbob_history abipop

0.8 abipop abipop_bbob_history_best

0.6 0.4

best 2009 best 2009

0.2 0.00

1

abipop_bbob abipop

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

Figure 2: Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension for 50 targets in 10[−8..2] for all functions and subgroups in 5-D. The “best 2009” line corresponds to the best ERT observed during BBOB 2009 for each single target.

3

separable fcts

moderate fcts

Proportion of function+target pairs

weakly structured multi-modal fcts 1.0 f20-24,20-D best 2009 best 2009 0.8 abipop abipop_bbob

0.6

Proportion of function+target pairs

Proportion of function+target pairs

ill-conditioned fcts

multi-modal fcts 1.0 f15-19,20-D 0.8

0.4

0.00

1

abipop_bbob abipop

2 3 4 5 6 7 8 log10 of (# f-evals / dimension) all functions

1.0 f15-24,20-D

best 2009 best 2009

0.8 abipop abipop_bbob

0.6

0.2

0.2

abipop_bbob_history_best abipop 0.0

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

abipop abipop_bbob_history

0.2

0.4 abipop abipop_bbob_history

1

abipop abipop_bbob_history_best

0.6

0.4

0.00

best 2009 best 2009

0

abipop abipop_bbob_history

1

abipop_bbob_history_best abipop

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

Figure 3: Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension for 50 targets in 10[−8..2] for all functions and subgroups in 20-D. The “best 2009” line corresponds to the best ERT observed during BBOB 2009 for each single target.

4

Table 1: ERT on f1 in 5-D over ERTbest obtained in BBOB 2009

Table 2: ERT on f2 in 5-D over ERTbest obtained in BBOB 2009

Table 3: ERT on f3 in 5-D over ERTbest obtained in BBOB 2009

Table 4: ERT on f4 in 5-D over ERTbest obtained in BBOB 2009

Table 5: ERT on f5 in 5-D over ERTbest obtained in BBOB 2009

Table 6: ERT on f6 in 5-D over ERTbest obtained in BBOB 2009

Table 7: ERT on f7 in 5-D over ERTbest obtained in BBOB 2009

Table 8: ERT on f8 in 5-D over ERTbest obtained in BBOB 2009

Table 9: ERT on f9 in 5-D over ERTbest obtained in BBOB 2009

Table 10: ERT on f10 in 5-D over ERTbest obtained in BBOB 2009

Table 11: ERT on f11 in 5-D over ERTbest obtained in BBOB 2009

Table 12: ERT on f12 in 5-D over ERTbest obtained in BBOB 2009

5

Table 13: ERT on f13 in 5-D over ERTbest obtained in BBOB 2009 Table 14: ERT on f14 in 5-D over ERTbest obtained in BBOB 2009 Table 15: ERT on f15 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f15 511 abipop 2.3(3) abipop 2.1(3) abipop1.9(1)

1e0 9310 2.4(2) 1.3(2) 1.9(2)

1e-1 19369 3.3(3) 1.8(1) 2.0(1)

1e-3 20073 3.2(3) 1.7(1) 2.0(1)

1e-5 20769 3.1(3) 1.7(1) 2.0(1)

1e-7 21359 3.1(3) 1.7(1) 2.0(1)

#succ 14/15 15/15 15/15 15/15

Table 16: ERT on f16 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f16 120 abipop1.5(1.0) abipop 1.7(1) abipop 2.9(2)

1e0 612 6.2(9) 3.9(5) 3.4(3)

1e-1 2662 2.7(2) 2.0(2) 4.1(5)

1e-3 10449 1.6(1) 1.5(1) 1.7(1)

1e-5 11644 1.7(1) 1.5(1) 1.6(1)

1e-7 12095 1.7(1) 1.4(1) 1.5(1)

#succ 15/15 15/15 15/15 15/15

Table 17: ERT on f17 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f17 5.2 abipop2.7(3) abipop 3.3(5) abipop2.7(3)

1e0 215 3.9(0.9) 1.1(0.7) 1.1(0.6)

1e-1 899 2.6(6) 1.4(0.2) 1.2(1)

1e-3 3669 2.7(3) 1.5(2) 1.3(1)

1e-5 6351 2.7(2) 2.8(2) 1.8(2)

1e-7 7934 3.0(2) 3.8(1) 2.4(2)

#succ 15/15 15/15 15/15 15/15

Table 18: ERT on f18 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f18 103 abipop1.2(0.7) abipop 1.4(0.8) abipop 1.3(0.9)

1e0 378 2.5(0.4) 3.3(7) 1.2(0.3)

1e-1 3968 1.4(1) 1.6(2) 2.0(3)

1e-3 9280 1.7(2) 1.7(1) 2.4(1)

1e-5 10905 2.9(2) 2.1(1) 2.5(0.8)

1e-7 12469 3.3(1) 2.3(1) 2.3(0.7)

#succ 15/15 15/15 15/15 15/15

Table 19: ERT on f19 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f19 1 abipop 23(34) abipop19(18) abipop 24(20)

1e0 1 3351(4964) 2723(3170) 2451(1722)

1e-1 242 129(109) 171(136) 130(133)

1e-3 1.2e5 1.6(0.9) 1.5(1) 1.0(0.9)

1e-5 1.2e5 1.6(0.9) 1.5(1) 1.0(0.9)

1e-7 1.2e5 1.6(0.9) 1.5(1) 1.0(0.9)

#succ 15/15 15/15 15/15 15/15

Table 20: ERT on f20 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f20 16 abipop 4.2(3) abipop3.8(3) abipop 4.7(3)

1e0 851 13(16) 12(11) 9.0(8)

1e-1 38111 3.4(3) 2.2(1) 2.4(2)

1e-3 54470 2.6(2) 1.6(0.8) 1.8(1)

6

1e-5 54861 2.6(2) 1.6(0.8) 1.8(1)

1e-7 55313 2.6(2) 1.6(0.8) 1.8(1)

#succ 14/15 15/15 15/15 15/15

Table 21: ERT on f21 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f21 41 abipop 2.1(2) abipop 1.6(1) abipop1.6(2)

1e0 1157 3.8(5) 210(289) 144(160)

1e-1 1674 5.1(7) 364(419) 399(525)

1e-3 1705 8.8(7) 357(424) 392(482)

1e-5 1729 8.8(7) 352(431) 387(476)

1e-7 1757 8.7(7) 347(432) 381(524)

#succ 14/15 15/15 12/15 11/15

Table 22: ERT on f22 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f22 71 abipop 7.6(11) abipop489(46) abipop159(337)

1e0 386 15(18) 659(884) 296(237)

1e-1 938 82(129) 2075(2661) 256(431)

1e-3 1008 82(160) 1931(2406) 238(401)

1e-5 1040 81(159) 1872(2364) 231(388)

1e-7 1068 79(155) 1824(2299) 226(378)

#succ 14/15 15/15 6/15 14/15

Table 23: ERT on f23 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f23 3.0 abipop2.0(2) abipop 2.2(2) abipop 2.3(2)

1e0 518 9.4(10) 13(9) 9.4(10)

1e-1 14249 2.7(4) 2.5(2) 1.7(2)

1e-3 31654 1.7(2) 1.4(1.0) 1.2(0.9)

1e-5 33030 1.7(2) 1.4(0.9) 1.1(0.9)

1e-7 34256 1.6(2) 1.3(0.9) 1.1(0.9)

#succ 15/15 15/15 15/15 15/15

Table 24: ERT on f24 in 5-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f24 1622 abipop 2.1(2) abipop1.7(1) abipop 2.8(4)

1e0 2.2e5 0.58(0.8) 8.2(11) 14(18)

1e-1 6.4e6 ∞ 0.33(0.4) 0.52(0.6)

1e-3 9.6e6 ∞ 0.31(0.3) 0.49(0.5)

7

1e-5 1.3e7 ∞ 0.39(0.4) 0.37(0.5)

1e-7 1.3e7 ∞ 1e6 0.39(0.5) 0.37(0.4)

#succ 3/15 0/15 3/15 3/15

Table 25: ERT on f1 in 20-D over ERTbest obtained in BBOB 2009

Table 26: ERT on f2 in 20-D over ERTbest obtained in BBOB 2009

Table 27: ERT on f3 in 20-D over ERTbest obtained in BBOB 2009

Table 28: ERT on f4 in 20-D over ERTbest obtained in BBOB 2009

Table 29: ERT on f5 in 20-D over ERTbest obtained in BBOB 2009

Table 30: ERT on f6 in 20-D over ERTbest obtained in BBOB 2009

Table 31: ERT on f7 in 20-D over ERTbest obtained in BBOB 2009

Table 32: ERT on f8 in 20-D over ERTbest obtained in BBOB 2009

Table 33: ERT on f9 in 20-D over ERTbest obtained in BBOB 2009

Table 34: ERT on f10 in 20-D over ERTbest obtained in BBOB 2009

Table 35: ERT on f11 in 20-D over ERTbest obtained in BBOB 2009

Table 36: ERT on f12 in 20-D over ERTbest obtained in BBOB 2009

8

Table 37: ERT on f13 in 20-D over ERTbest obtained in BBOB 2009 Table 38: ERT on f14 in 20-D over ERTbest obtained in BBOB 2009 Table 39: ERT on f15 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f15 30378 abipop 2.5(2) abipop 1.9(1) abipop1.8(0.6)

1e0 1.5e5 6.4(8) 6.7(4) 4.8(3)

1e-1 3.1e5 6.2(5) 4.9(4) 3.9(3)

1e-3 3.2e5 6.1(7) 4.8(4) 3.9(3)

1e-5 4.5e5 4.3(5) 3.5(3) 2.8(2)

1e-7 4.6e5 4.3(4) 3.4(3) 2.7(2)

#succ 15/15 11/15 12/15 13/15

Table 40: ERT on f16 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f16 1384 abipop 15(33) abipop 17(33) abipop14(24)

1e0 27265 1.2(2) 1.9(2) 1.3(2)

1e-1 77015 3.0(2) 3.9(4) 2.6(2)

1e-3 1.9e5 3.0(2) 8.8(8) 8.3(9)

1e-5 2.0e5 3.6(2) 12(11) 11(13)

1e-7 2.2e5 3.3(2) 13(14) 10(12)

#succ 15/15 15/15 10/15 10/15

Table 41: ERT on f17 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f17 63 abipop 1.8(2) abipop 1.6(1) abipop1.5(1)

1e0 1030 1.1(0.2) 1.1(0.3) 1.1(0.1)

1e-1 4005 6.4(12) 3.1(3) 6.3(17)

1e-3 30677 6.3(3) 4.7(4) 6.0(4)

1e-5 56288 5.7(2) 6.1(3) 7.0(2)

1e-7 80472 7.0(4) 6.2(2) 6.0(3)

#succ 15/15 15/15 15/15 15/15

Table 42: ERT on f18 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f18 621 abipop 1.3(0.2) abipop1.2(0.4) abipop 1.2(0.3)

1e0 3972 2.4(6) 5.4(9) 3.1(6)

1e-1 19561 6.4(6) 8.0(8) 5.6(5)

1e-3 67569 11(6) 6.1(3) 8.5(3)

1e-5 1.3e5 6.6(3) 3.8(2) 5.1(1)

1e-7 1.5e5 6.2(2) 3.6(1) 4.8(1)

#succ 15/15 14/15 15/15 14/15

Table 43: ERT on f19 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f19 1 abipop190(107) abipop 205(100) abipop 210(85)

1e0 1e-1 1 3.4e5 3.9e4(4e4) 3.0(2) 2.9e4(3e4) 2.1(1.0) 3.5e4(5e4) 2.0(0.7)

1e-3 6.2e6 12(13) 1.5(1) 0.82(0.5)

1e-5 6.7e6 11(11) 1.7(2) 0.86(0.7)

1e-7 6.7e6 11(12) 1.7(1) 0.86(0.6)

#succ 15/15 1/15 6/15 10/15

Table 44: ERT on f20 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f20 82 abipop 5.9(1) abipop 5.8(1) abipop5.7(2)

1e0 46150 5.6(2) 5.5(2) 4.9(2)

1e-1 3.1e6 ∞ 1.2(0.8) 1.3(0.8)

1e-3 5.5e6 ∞ 1.7(2) 3.5(3)

9

1e-5 5.6e6 ∞ 1.7(2) 3.5(4)

1e-7 5.6e6 ∞ 3e6 1.7(2) 3.4(4)

#succ 14/15 0/15 4/15 2/15

Table 45: ERT on f21 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f21 561 2.4(5) abipop abipop1164(1844) abipop1903(2460)

1e0 6541 43(46) 536(675) 509(635)

1e-1 14103 35(54) 249(318) 345(437)

1e-3 14643 34(52) 240(270) 332(398)

1e-5 15567 32(50) 225(273) 312(360)

1e-7 17589 28(44) 200(240) 276(341)

#succ 15/15 13/15 4/15 3/15

Table 46: ERT on f22 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f22 467 abipop 10(5) abipop1327(2572) abipop 301(1014)

1e0 5580 15(9) 904(1128) 580(737)

1e-1 23491 203(242) ∞ ∞

1e-3 24948 196(232) ∞ ∞

1e-5 26847 182(211) ∞ ∞

1e-7 1.3e5 36(43) ∞ 1e6 ∞ 1e6

#succ 12/15 4/15 0/15 0/15

Table 47: ERT on f23 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f23 3.2 abipop 2.1(2) abipop1.7(1) abipop 2.5(2)

1e0 1614 42(26) 54(32) 41(40)

1e-1 67457 5.0(4) 5.2(4) 4.8(4)

1e-3 4.9e5 16(14) 47(49) 14(14)

1e-5 8.1e5 11(10) 28(30) 8.5(9)

1e-7 8.4e5 11(9) 28(29) 8.2(8)

#succ 15/15 9/15 4/15 10/15

Table 48: ERT on f24 in 20-D over ERTbest obtained in BBOB 2009 ∆fopt 1e1 f24 1.3e6 abipop 0.28(0.3) abipop 5.4(7) abipop17(20)

1e0 7.5e6 1.3(1) 1.1(1) 3.2(4)

1e-1 5.2e7 ∞? ∞ ∞

1e-3 5.2e7 ∞? ∞ ∞

10

1e-5 5.2e7 ∞? ∞ ∞

1e-7 5.2e7 ∞ 4e6 ? ∞ 3e6 ∞ 3e6

#succ 3/15 0/15 0/15 0/15