Define a set of named features and logic to manage their dependencies.

[albu-diku]

Summary:

This work povides a way to make use of per-feature requirements files while maintaining the ability to have certain requirements overridden during build and other be met by system packages. This would make it suitable for immediate use in docker-migrid where it would replace the series of hand rolled pip install lines gated by feature checks and the package installation lines those checks contain.

Detail:

The codebase has a number of features that, when selected, require certain dependencies. Thus far the handling for this has been split between this repository and the docker-migrid, which encodes a large amount of version information for the various features and in addition allows overriding these at build time.

There is a desire to see these become per-feature requirements files expressed in this repository - that would mean the dependencies are captured alongside the code that depends upon them. However, the ability to override the versions being used and, more critically, to make use of packages provided by the distribution must remain possible.

This PR introduces work that will allow that. The features are explicitly defined by name in a static file that describes them. The added features tool reads this file and, after consulting a source for what is enabled (this can be the environment but also a .env file, as used within the docker builds, directly) will output the lines needed to install the packages.

This simple conceptual design allows us to handle package overrides and distribution packages. We maintain a list of environmental keys to be consulted for certain package versions and if present expand the packages we must install from the requirements file and make use of the an applicable override instead. In the situation of distribution packages, we perform the same requirements expansion but check whether the dependency is already met in the python environment and skip its installation if so.

Review

To anyone taking a look: not all the features are fully defined here as a way to first share the approach. The tests coves all installation cases, but when run directly the repository level notion of defined features are used. Two named features re currently defined, CLOUD and MIGUX (which likely gives you an idea of what I was working on). I would recommend comparing the output from the following commands:

./envhelp/lpython ./mig/install/features.py install --env --check
ENABLE_CLOUD=true ./envhelp/lpython ./mig/install/features.py install --env --check
ENABLE_CLOUD=true OPENSTACKSDK_VERSION_OVERRIDE=1.2.3 ./envhelp/lpython ./mig/install/features.py install --env --check