Getting started—Analysing and comparing results (postprocessing)
Benchmarking performance results are displayed with the cocopp
Python module and a web browser.
Installation (assuming Python is present)
From a system shell, execute
python -m pip install cocopp
General usage
The cocopp
“postprocessing” allows basic usage in a system shell:
python -m cocopp name_1 [name_2 [name_3 [... ]]]
creates a couple of hundred figures and (after some seconds or a minute) displays the page ppdata/index.html
with the default browser. The arguments name_1
, name_2
,… ([]
-braces indicate optional arguments) are either local folders containing COCO data from a single benchmarked algorithm (typically like exdata/algorithm_name
) or names of benchmarked algorithms from the COCO data archive (see below).
The recommended usage is however from an ipython
shell or a Jupyter notebook via the cocopp.main
method.
>>> import cocopp
>>> cocopp.main('name_1 [name_2 [name_3 ...]]')
gives the same result as the above.
Finding data to compare with
Hundreds of COCO comparative data sets are provided online and can be listed, filtered, and retrieved via cocopp.archives
and processed alone or together with local data. For example
>>> import cocopp
>>> cocopp.archives.bbob('bfgs') # list COCO data of bfgs on the bbob suite
'2009/BFGS_ros_noiseless.tgz',
['2012/DE-BFGS_voglis_noiseless.tgz',
'2012/PSO-BFGS_voglis_noiseless.tgz',
'2014-others/BFGS-scipy_Baudis.tgz',
'2014-others/L-BFGS-B-scipy_Baudis.tgz',
'2018/BFGS-M-17_Blelly.tgz',
'2018/BFGS-P-09_Blelly.tgz',
'2018/BFGS-P-Instances_Blelly.tgz',
'2018/BFGS-P-StPt_Blelly.tgz',
'2018/BFGS-P-range_Blelly.tgz',
'2019/BFGS-scipy-2019_Varelas.tgz',
'2019/L-BFGS-B-scipy-2019_Varelas.tgz']
lists all data sets run on the bbob
testbed containing 'bfgs'
in their name.
Examples
The cocopp.main
function takes any number of strings separated by spaces as argument (or a list of strings). For example, the first three entries from the above list can be postprocessed like
>>> dsl = cocopp.main('bfgs! 12/DE-BFGS 12/PSO-BFGS') # "!" picks the first match only
Each string must give a single unique match (exceptions are specified below), where the exclamation mark indicates the first match in the results from cocopp.archives.all('bfgs')
. Otherwise, an (informative) ValueError
is raised. The order can be changed and determines the colors in the figures. This call creates a few hundreds of figures and modifies and opens ppdata/index.html
in a browser, and finally returns a data set list of all read-in data. We can also use regular expression syntax. For example to display all BFGS data from the bbob
suite,
>>> dsl = cocopp.main('bbob/.*bfgs')
where .*
matches any character any number of times (including zero) and the expression must match the beginning of the archive entry name.
As a special case, a trailing *
indicates all names containing the preceding substring (not a regular expression). The output figures of
>>> dsl = cocopp.main('bob/2009/*') # may take some more time when these data are downloaded
contains 31 algorithms and can be browsed at https://numbbo.github.io/ppdata-archive/bbob/2009. To display algorithms in the background, the cocopp.genericsettings.background
variable needs to be set:
>>> cocopp.genericsettings.background = {None: cocopp.archives.bbob.get_extended('2009/*')} # may take some time when these data are downloaded
where None
invokes the default color and line style (solid grey, cocopp.genericsettings.background_default_style
).
Finally, we can, for example, compare our own data with a version of the Nelder-Mead simplex downhill algorithm, here the first 'nelder'
-matching archived algorithm, while all other COCO data from 2009 (on the bbob
suite) are shown in the background:
>>> dsl = cocopp.main('exdata/my_data nelder!')