Instance repetitions are not the same

[nikohansen]

Related to a recent commit trying to fix the number of instance repetitions in a DataSet.

I did not run (yet) the profiling itself, but with the newest code, I get new warnings such as:

C:\Users\brockhof\AppData\Local\anaconda3\envs\coco-web\Lib\site-packages\cocopp\pproc.py:2160: UserWarning: Instance counts should be the same but they are not.
lcm(instance_counters) = lcm(Counter({1: 18, 2: 18, 3: 18, 4: 18, 5: 17})) = 306 set to 18
  warnings.warn("Instance counts should be the same but they are not.\n"
C:\Users\brockhof\AppData\Local\anaconda3\envs\coco-web\Lib\site-packages\cocopp\pproc.py:2160: UserWarning: Instance counts should be the same but they are not.
lcm(instance_counters) = lcm(Counter({1: 21, 2: 21, 3: 20, 4: 20, 5: 20})) = 420 set to 21
  warnings.warn("Instance counts should be the same but they are not.\n"
C:\Users\brockhof\AppData\Local\anaconda3\envs\coco-web\Lib\site-packages\cocopp\pproc.py:2160: UserWarning: Instance counts should be the same but they are not.
lcm(instance_counters) = lcm(Counter({1: 19, 2: 18, 3: 18, 4: 18, 5: 18})) = 342 set to 19
  warnings.warn("Instance counts should be the same but they are not.\n"
C:\Users\brockhof\AppData\Local\anaconda3\envs\coco-web\Lib\site-packages\cocopp\pproc.py:2160: UserWarning: Instance counts should be the same but they are not.
lcm(instance_counters) = lcm(Counter({1: 29, 2: 29, 3: 29, 4: 28, 5: 28})) = 812 set to 29
  warnings.warn("Instance counts should be the same but they are not.\n"

[overall about 20 of them - the last entry(s) being always the one with the smallest number of repetitions]

Originally posted by @brockho in #60

[brockho]

Happens with this data set.

[nikohansen]

This looks like a genuine reflection of the data which seem to have some last runs missing. For example bbobexp_f25.info misses the last instance in dimension three

[...]
data_f25/bbobexp_f25_DIM3-0016.dat, 1:13186|1.5e-04, 2:17627|1.2e-03, 3:8587|2.6e-03, 4:17459|8.9e-04, 5:9595|1.4e-03
suite = 'bbob-constrained', funcId = 25, DIM = 5, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f25/bbobexp_f25_DIM5-0016.dat, 1:22537|1.3e-02, 2:15068|1.3e-04, 3:14340|1.9e-03, 4:15050|4.9e-04, 5:19398|1.1e-02
suite = 'bbob-constrained', funcId = 25, DIM = 10, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f25/bbobexp_f25_DIM10-0016.dat, 1:22045|1.4e-03, 2:22045|6.5e-03, 3:22046|5.5e-03, 4:22045|1.1e-02, 5:22045|3.8e-02
suite = 'bbob-constrained', funcId = 25, DIM = 20, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f25/bbobexp_f25_DIM20-0016.dat, 1:21710|1.8e-02, 2:21712|4.5e-02, 3:21712|3.3e-02, 4:21711|8.2e-03, 5:21712|1.1e-02
suite = 'bbob-constrained', funcId = 25, DIM = 3, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f25/bbobexp_f25_DIM3-0017.dat, 1:6307|5.7e-06, 2:16899|1.1e-03, 3:8267|2.5e-03, 4:22881|5.8e-04

and bbobexp_f53.info misses the last three instances in dimension 3

suite = 'bbob-constrained', funcId = 53, DIM = 20, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM20-0018.dat, 1:21711|8.6e+02, 2:21712|1.4e+03, 3:21710|1.5e+03, 4:21710|1.4e+03, 5:21711|8.4e+02
suite = 'bbob-constrained', funcId = 53, DIM = 2, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM2-0019.dat, 1:5883|1.4e+02, 2:7365|2.2e+02, 3:6541|6.4e+02, 4:5686|3.9e+02, 5:8557|8.9e+01
suite = 'bbob-constrained', funcId = 53, DIM = 3, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM3-0019.dat, 1:14475|1.2e+03, 2:9139|6.8e+02, 3:7962|3.0e+02, 4:6130|8.5e+02, 5:8931|2.9e+02
suite = 'bbob-constrained', funcId = 53, DIM = 5, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM5-0019.dat, 1:11225|3.6e+02, 2:20730|1.8e+02, 3:18298|1.9e+03, 4:17463|3.0e+02, 5:20010|5.6e+02
suite = 'bbob-constrained', funcId = 53, DIM = 10, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM10-0019.dat, 1:17393|5.3e+02, 2:22045|2.0e+03, 3:18943|1.1e+03, 4:22044|7.9e+02, 5:22045|7.3e+02
suite = 'bbob-constrained', funcId = 53, DIM = 20, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM20-0019.dat, 1:21710|1.5e+03, 2:21710|1.4e+03, 3:21710|1.7e+03, 4:21710|1.4e+03, 5:21710|1.1e+03
suite = 'bbob-constrained', funcId = 53, DIM = 2, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM2-0020.dat, 1:9537|1.4e+02, 2:4509|2.1e+02, 3:8781|7.4e+02, 4:5512|6.1e+02, 5:4538|6.8e-13
suite = 'bbob-constrained', funcId = 53, DIM = 3, Precision = 1.000e-08, algId = 'comocma.como.Sofomore', coco_version = '2.8.1', logger = 'bbob-constrained', data_format = 'bbob-new2', settings = ''
% 
data_f53/bbobexp_f53_DIM3-0020.dat, 1:7699|1.7e+02, 2:11707|9.9e+02

and all these are also missing in the data file.

Could this be some sort of timeout issue when running the experiment?

[brockho]

Could this be some sort of timeout issue when running the experiment?

Yes, that's totally possible. In particular because it happens in the last line of the .info file, this is the most likely explanation. I will re-run the experiment to investigate this further.

[nikohansen]

Though, timeout issues are more likely to happen in larger dimension, not sure if my selection was biased to the smaller dimension.