Getting started—Analysing and comparing results (postprocessing)

Benchmarking performance results are displayed with the cocopp Python module and a web browser.

Installation (assuming Python is present)

From a system shell, execute

python -m pip install cocopp

General usage

The cocopp “postprocessing” allows basic usage in a system shell:

python -m cocopp name_1 [name_2 [name_3 [... ]]]

creates a couple of hundred figures and (after some seconds or a minute) displays the page ppdata/index.html with the default browser. The arguments name_1, name_2,… ([]-braces indicate optional arguments) are either local folders containing COCO data from a single benchmarked algorithm (typically like exdata/algorithm_name) or names of benchmarked algorithms from the COCO data archive (see below).

The recommended usage is however from an ipython shell or a Jupyter notebook via the cocopp.main method.

>>> import cocopp
>>> cocopp.main('name_1 [name_2 [name_3 ...]]')

gives the same result as the above.

Finding data to compare with

Hundreds of COCO comparative data sets are provided online and can be listed, filtered, and retrieved via cocopp.archives and processed alone or together with local data. For example

>>> import cocopp
>>> cocopp.archives.bbob('bfgs')  # list COCO data of bfgs on the bbob suite

['2009/BFGS_ros_noiseless.tgz',
 '2012/DE-BFGS_voglis_noiseless.tgz',
 '2012/PSO-BFGS_voglis_noiseless.tgz',
 '2014-others/BFGS-scipy_Baudis.tgz',
 '2014-others/L-BFGS-B-scipy_Baudis.tgz',
 '2018/BFGS-M-17_Blelly.tgz',
 '2018/BFGS-P-09_Blelly.tgz',
 '2018/BFGS-P-Instances_Blelly.tgz',
 '2018/BFGS-P-StPt_Blelly.tgz',
 '2018/BFGS-P-range_Blelly.tgz',
 '2019/BFGS-scipy-2019_Varelas.tgz',
 '2019/L-BFGS-B-scipy-2019_Varelas.tgz']

lists all data sets run on the bbob testbed containing 'bfgs' in their name.

Examples

The cocopp.main function takes any number of strings separated by spaces as argument (or a list of strings). For example, the first three entries from the above list can be postprocessed like

>>> dsl = cocopp.main('bfgs! 12/DE-BFGS 12/PSO-BFGS')  # "!" picks the first match only

Each string must give a single unique match (exceptions are specified below), where the exclamation mark indicates the first match in the results from cocopp.archives.all('bfgs'). Otherwise, an (informative) ValueError is raised. The order can be changed and determines the colors in the figures. This call creates a few hundreds of figures and modifies and opens ppdata/index.html in a browser, and finally returns a data set list of all read-in data. We can also use regular expression syntax. For example to display all BFGS data from the bbob suite,

>>> dsl = cocopp.main('bbob/.*bfgs')

where .* matches any character any number of times (including zero) and the expression must match the beginning of the archive entry name.

As a special case, a trailing * indicates all names containing the preceding substring (not a regular expression). The output figures of

>>> dsl = cocopp.main('bob/2009/*')  # may take some more time when these data are downloaded

contains 31 algorithms and can be browsed at https://numbbo.github.io/ppdata-archive/bbob/2009. To display algorithms in the background, the cocopp.genericsettings.background variable needs to be set:

>>> cocopp.genericsettings.background = {None: cocopp.archives.bbob.get_extended('2009/*')}  # may take some time when these data are downloaded

where None invokes the default color and line style (solid grey, cocopp.genericsettings.background_default_style).

Finally, we can, for example, compare our own data with a version of the Nelder-Mead simplex downhill algorithm, here the first 'nelder'-matching archived algorithm, while all other COCO data from 2009 (on the bbob suite) are shown in the background:

>>> dsl = cocopp.main('exdata/my_data nelder!')