Consider moving datasets out of the core python library

[tpgillam]

I very much appreciate that moocore has a narrow set of functionality with minimal dependencies.

I'm wondering if the _datasets module, and corresponding data directory, could or should be moved into the examples, and hence not bundled in the wheel. On MacOS at least, this would remove between 1/4 and 1/3 of the installed size-on-disk. For most practical uses of this package that I can imagine, end-users will be solving their own problems rather than examples.

Also, moocore's dependency tree is currently:

├── moocore v0.1.10
│   ├── cffi v1.17.1
│   │   └── pycparser v2.22
│   ├── numpy v2.3.2
│   └── platformdirs v4.3.8

The platformdirs package is used in exactly one place, for caching datasets in _datasets. So this dependency can also be removed if removing datasets.

[MLopez-Ibanez]

Now that we can download and cache datasets, I would like to remove the large datasets from the source code, starting from those already available in the testsuite repository. But they are still useful for examples, testing and benchmarking. Is the size of moocore currently a problem?

About removing platformdirs. We only use a tiny part of its functionality so if someone sends a PR implementing that part and getting rid of it, I'll be happy to remove it. Is platformdirs causing any problems?

[tpgillam]

Probably it's less about the size, and more about conceptual simplicity - just a thought though!

I see this as a valuable feature when positioning moocore as more of a low-level library, upon which higher level optimisation frameworks (like pymoo) would depend.

[MLopez-Ibanez]

I have thought about this and I would remove any datasets that are only used in tests and benchmarks. However, for datasets used in examples and doctests, I cannot think of a more convenient way to provide them so that users can run the examples themselves. Maybe some of them can be replaced with randomly generated ones.

Nevertheless, the next release will compress the larger datasets and also the size of the dynamic library, which is the largest part of the installation, has been reduced by 15%.

[tpgillam]

I cannot think of a more convenient way to provide them so that users can run the examples themselves

It's a good point!

moocore doesn't distribute examples in the wheel. The documentation for the examples does already say that additional dependencies may be required, e.g. for plotting. So, possibly datasets could be moved to a separate package... I could actually see a moo-problems module that aggregates commonly used toy, benchmark, and awkward test cases, that's independent of any particular way of solving them, being quite desirable.

Or possibly suitable datasets are available in other already published modules, e.g. sklearn.datasets.

Anyway, just ideas!