Postprocessing is extermely slow for large numbers of repetitions

[brockho]

When running the postprocessing on data, produced on the bbob-constrained suite with the latest cocoex code (2.8.1), a large(r) number of repetitions is envoked for the algorithm, I ran (e.g. about 25 on $f_7$). This causes the postprocessing to be extremely slow: postprocessing two of those large-repetition algorithms together with some from the data archive takes >5 hours to run, in particular the aggregated ECDFs take in the order of 30+ minutes per plot.

The output warns me about this:

[...]
  Will generate output data in folder ppdata\constrained_10000_10kRu_AL1-C_AL2-C_AL3-C_AL4-C_COBYL_et_al_101609h1557
    this might take several minutes.
ECDF graphs per noise group...
C:\Users\brockhof\AppData\Local\anaconda3\Lib\site-packages\cocopp\compall\pprldmany.py:700: UserWarning: Sample size equals 19530 which may take very long. This is likely to be unintended, hence a bug.
  warnings.warn(warntxt)
  done (Thu Oct 16 12:50:23 2025).
ECDF graphs per function group...
C:\Users\brockhof\AppData\Local\anaconda3\Lib\site-packages\cocopp\compall\pprldmany.py:700: UserWarning: Sample size equals 15960 which may take very long. This is likely to be unintended, hence a bug.
  warnings.warn(warntxt)
C:\Users\brockhof\AppData\Local\anaconda3\Lib\site-packages\cocopp\compall\pprldmany.py:700: UserWarning: Sample size equals 19530 which may take very long. This is likely to be unintended, hence a bug.
  warnings.warn(warntxt)
C:\Users\brockhof\AppData\Local\anaconda3\Lib\site-packages\cocopp\compall\pprldmany.py:700: UserWarning: Sample size equals 11280 which may take very long. This is likely to be unintended, hence a bug.
  warnings.warn(warntxt)
C:\Users\brockhof\AppData\Local\anaconda3\Lib\site-packages\cocopp\compall\pprldmany.py:700: UserWarning: Sample size equals 19530 which may take very long. This is likely to be unintended, hence a bug.
  warnings.warn(warntxt)
[...]

Together with @nikohansen, we nailed down the origin of the problem in the experiment itself where the large number of possible repetitions was 100 in my case (the default in the example experiment in python). In any case, the postprocessing should be able to deal with it---hence, I opened the issue here.

[nikohansen]

It should be worth trying to run

python -m cocopp.reduce_size folder_name

on the folder with the experimental data and see whether this resolves the issue. See also numbbo/coco-experiment#48 (comment)

[brockho]

Sorry for the late reply, but I looked into this closer now.

First of all, the total comparison times of the postprocessing on my machine (everything only roughly) for single algorithms:

• 2022/COBYLA took ~9 minutes
• the original exdata folder from above took ~21 minutes (i.e. ~2.5 times longer than COBYLA)
• the reduced_size, suggested by @nikohansen was way smaller (72MB vs. 525MB), but still took ~21 minutes

I chose 2022/COBYLA as reference, since the "problematic" data is on the bbob-constrained suite which produces more plots and has also more functions than the classical bbob (bipop! takes "only" about 2 minutes to postprocess). Note that I had all data already on my disk, so nothing had to be downloaded and/or extracted).

I then did, as suggested by @nikohansen offline, a profiling of the entire postprocessing call. The top entries from cprofile.run(...) for the "problematic" data are those ones:

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
57966	688.438	0.012	713.297	0.012	toolsstats.py:212(drawSP)
403800/734	55.901	0	0.331	0	lines.py:758(draw)
157739	54.05	0	86.185	0.001	pproc.py:1420(maxfgevals)
1862753	30.647	0	44.224	0	scale.py:233(transform_non_affine)
12000613/4215763	30.355	0	72.623	0	{built-in method builtins.sum}
1729362/901512	22.304	0	65.027	0	transforms.py:2217(transform_non_affine)
4083/4	20.177	0.005	69.949	17.487	backend_agg.py:206(get_text_width_height_descent)
557/1	17.321	0.031	0.059	0.059	backend_agg.py:185(draw_text)
80306	16.394	0	36.573	0	backend_agg.py:93(draw_path)
2456978	12.569	0	12.569	0	{method 'reduce' of 'numpy.ufunc'

On the contrary, the first entries for COBYLA are those:

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
458954/929	61.705	0	0.303	0	lines.py:758(draw)
2142680	34.98	0	50.595	0	scale.py:233(transform_non_affine)
1924987/1006671	24.545	0	72.158	0	transforms.py:2217(transform_non_affine)
4467/4	21.669	0.005	98.507	24.627	backend_agg.py:206(get_text_width_height_descent)
619/1	19.651	0.032	0.07	0.07	backend_agg.py:185(draw_text)
92436	16.859	0	39.18	0	backend_agg.py:93(draw_path)
2769118	14.055	0	14.055	0	{method 'reduce' of 'numpy.ufunc' objects}
681873	13.018	0	32.414	0	_function_base_impl.py:5473(insert)
26424/7	12.754	0	0.004	0.001	_text_helpers.py:46(layout)
5968911/5820638	12.245	0	33.705	0	{built-in method builtins.next}

The toolsstats.py:212(drawSP) line, comes only a bit later at position 48:

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
73158	4.838	0	10.718	0	toolsstats.py:212(drawSP)

[nikohansen]

Looks like it could be an oversight when computing the number of samples for bootstrapping with a diverse number of different repetitions.

The data size remains a concern too, that is, a proper logging behavior.

[nikohansen]

Can you check whether the problem persists when setting

cocopp.genericsettings.simulated_runlength_bootstrap_sample_size = 25

say, with the reduced data size? EDIT: this is not likely to make a difference, as it is by default already small-ish and probably not even finally used.

[brockho]

Unfortunately, nothing really changed (run with cocopp.genericsettings.simulated_runlength_bootstrap_sample_size = 25 and for the reduced folder size):

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
57966	642.373	0.011	666.586	0.011	toolsstats.py:212(drawSP)
404760/734	53.712	0	0.31	0	lines.py:758(draw)
157741	51.713	0	82.566	0.001	pproc.py:1420(maxfgevals)
1871885	29.877	0	43.106	0	scale.py:233(transform_non_affine)
12000447/4215597	29.11	0	69.718	0	{built-in method builtins.sum}
1733902/903790	21.44	0	62.931	0	transforms.py:2217(transform_non_affine)
4083/4	19.961	0.005	61.806	15.451	backend_agg.py:206(get_text_width_height_descent)
557/1	16.634	0.03	0.061	0.061	backend_agg.py:185(draw_text)
80312	15.65	0	34.941	0	backend_agg.py:93(draw_path)
2460358	12.324	0	12.324	0	{method 'reduce' of 'numpy.ufunc'}

[nikohansen]

Just FTR, we see above the warning

pprldmany.py:700: UserWarning: Sample size equals 19530 which may take very long. This is likely to be unintended, hence a bug.
  warnings.warn(warntxt)
  done (Thu Oct 16 12:50:23 2025).

which predicts the problem.

[nikohansen]

The relevant code snippet was line 683--700 in pprldmany.py

        # determine a good samplesize
        run_numbers = []
        for dsl in dictDim.values():
            run_numbers.extend([ds.nbRuns() for ds in dsl])
        if genericsettings.in_a_hurry >= 100:
            samplesize = max(run_numbers)
        else:
            try: lcm = np.lcm.reduce(run_numbers)  # lowest common multiplier
            except: lcm = max(run_numbers)  # fallback for old numpy versions
            # slight abuse of bootstrap_sample_size to avoid a huge number
            samplesize = min((int(genericsettings.simulated_runlength_bootstrap_sample_size), lcm))
        if testbedsettings.current_testbed.instances_are_uniform:
            samplesize = max((int(genericsettings.simulated_runlength_bootstrap_sample_size),
                              samplesize))  # maybe more bootstrapping with unsuccessful trials
        if samplesize > 1e4:
            warntxt = ("Sample size equals {} which may take very long. "
                       "This is likely to be unintended, hence a bug.".format(samplesize))
            warnings.warn(warntxt)

Given in_a_hurry > 100 (as it is by default), this means that at least one DataSet has 19530 entries (given I parse the code correctly)? This should never be the case, as the number of repetitions is limited to 100?

Maybe as a next step we should check the nbRuns() of the data in question.

[brockho]

I don't know the exact code, you wanted me to run because I initially did not find the nbruns() functionality, but I did run the following instead:

import cocopp
dsl = cocopp.load('MY_ALREADY_REDUCED_DATA_SET')
nbruns = []
for dd in dsl:
   nbruns.append(len(dd.funvals))
print(sorted(nbruns)[-11:])

and I get [131, 131, 133, 135, 137, 139, 142, 147, 149, 155, 155].

Then I tried to run the code snipped above and I did the following:

dictDim = dsl.dictByDim()
run_numbers = []
for dslbd in dictDim.values():
   run_numbers.extend([ds.nbRuns() for ds in dslbd])
print(sorted(run_numbers)[-11:])

resulting in [140, 140, 140, 140, 150, 150, 155, 155, 155, 170, 170].

Is this what you intended me to check, @nikohansen?

[brockho]

BTW, changing the simulated_runlength_bootstrap_sample_size from 100 to 25 does not change anything in the above numbers.

[nikohansen]

I am confused, I don't understand which path the code execution is taking and why. I can't figure out why we end up in the warning under if samplesize > 1e4: because we should have set samplesize = max(run_numbers). What am I missing?

[nikohansen]

@brockho If you provide me with the offending data (the small size should be good enough), I can have a close look.

[brockho]

I have temporarily put it on my homepage. Not sure that it matters what the algorithm does, but it is mainly COMO-CMA-ES with two kernels, optimizing f as first objective and sum(max(g_i(x),0)) as second objective. The data is already reduced in size, ~17MB.

[nikohansen]

I have temporarily put it on my homepage.

Processing these data took a little shy of 5 minutes on my Macbook. Most of the time is spent in the ECDF per function, ~4 minutes.

Tue Nov 25 16:16:19 2025
Post-processing (1)
  Using 1 data set:
    exdata-issue060/100000D-101622h0635

Post-processing (1)
  loading data...
[...]
UserWarning: Some DataSet of 100000D-101622h0635 was not consistent
  warnings.warn('Some DataSet of {0} was not consistent'.format(self.algId))  # should rather be in the previous messages
  Will generate output data in folder ppdata/100000D-101622h0635_112516h1619
    this might take several minutes.
Scaling figures...
  done (Tue Nov 25 16:16:54 2025).
Scaling wrt constraints...
  done (Tue Nov 25 16:17:14 2025).
Generating LaTeX tables...
  done (Tue Nov 25 16:17:19 2025).
ECDF graphs...
  done (Tue Nov 25 16:17:31 2025).
ECDF graphs per function...
  done (Tue Nov 25 16:21:13 2025).
ERT loss ratio figures and tables...
  done (Tue Nov 25 16:21:13 2025).
Output data written to folder [...]
Setting changes in `cocopp.genericsettings` compared to default:
    simulated_runlength_bootstrap_sample_size: from 30 to 10.098990100989901
    foreground_algorithm_list: from [] to ['exdata-issue060/1000...
ALL done (Tue Nov 25 16:21:13 2025).
Tue Nov 25 16:21:13 2025