Improve core metadata compliance for PKG-INFO

[abravalheri]

Summary of changes

• Add Requires-Dist to PKG-INFO files and make them compatible with .dist-info/METADATA
• This PR extracts PKG-INFO logic from setuptools.dist to setuptools._core_metadata.
• Improve 'atomicity' when writing .egg-info/PKG-INFO.
  • It seems that there is a race condition between importlib.metadata reading PKG-INFO and setuptools.build_meta writing to PKG-INFO, which causes importlib.metadata to emit a deprecation warnings (which will be converted in the future into an error).
  • I have observed in the past errors in the CI behaving in a flaky way, and I think those were associated to this race condition.

In the future the _core_metadata module can be used directly to generated .dist-info/METADATA.

---

This is part of a series of PRs:

• Avoid modification of install_requires and extra_requires that deviates from core metadata #3903
• Improve core metadata compliance for PKG-INFO #3904
• Avoid undocumented pypa/wheel API in dist_info #3905
• Implement a simplified wheel builder (avoid undocumented pypa/wheel API) #3906
• Avoid pypa/wheel API in editable_wheel #3907
• Adopt bdist_wheel from pypa/wheel #3908

The motivation for this series of PRs is the following:

• Logic for generating .egg-info and .dist-info directories is intertwined and implicit
  (See Adopt bdist_wheel from wheel project #1386).
• Setuptools uses pypa/wheel API which is not stable yet and is very likely to change in the future.
• pypa/wheel maintainers previously described that the long term vision is to transfer bdist_wheel
  directly to setuptools (See pypa/wheel#262, pypa/wheel#472, pypa/wheel#472).

---

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d/.
  (See documentation for details)

[abravalheri]

After some investigation and debugging it seems that the PKG-INFO file starts with the correct content, but ends up empty when the tests start running.

There might be a test that is not isolated modifying the PKG-INFO file directly...

This seems to be a flaky behaviour, and I am having trouble to replicate it locally, but I can see it in can some tests in the CI.

[abravalheri]

I noticed that when I improve the isolation of egg_info¹ used in setuptools.build_meta:get_requires*², setuptools starts to get trouble to bootstrap.
It may seem that it is relying on the fact that setuptools.build_meta:get_requires* will generate a setuptools.egg-info/PKG-INFO file in the project hook that will be picked up by importlib.metadata in the subsequent hook calls.

Footnotes

1. by setting egg_base to a temporary directory ↩

2. The isolation is necessary to avoid race conditions caused by importlib.metadata reading stuff from the .egg-info folder in the project root, while the .egg-info folder is being written. This situation seems to occur frequently because we use pytest-xdist, pytest-checkdocs, pytest-perf and other plugins that consume project metadata using build.util.project_wheel_metadata. I have been observing flaky tests because of this behaviour in the past months. ↩

[abravalheri]

If we try to further enhance egg_info isolation to avoid race conditions in the tests, and set up egg_base when packaging setuptools itself, the setuptools sdist will not contain the setuptools.egg-info folder... This means that it will not have entry_points.txt.

However some attributes in setup.cfg and setup.py are implemented via the distutils.setup_keyworks entry-point. Which means that they will not be available for being used and setting these attributes will cause errors.

[abravalheri]

It seems that an easier approach is not try to change where egg_info writes files for now (since this task is very difficult). Instead I tried to make write_pkg_info a more atomic operation.

[abravalheri]

@jaraco, would you be OK if I move this PR (#3904) and #3903 ahead?¹

The main idea of the 2 PR's is to improve PKG-INFO (right now install_requires and extras_require are mainly analysed by setuptools from a *.egg-info/requires.txt perspective, these 2 PRs shift the perspective to concern more with PKG-INFO).

Footnotes

1. I know that I presented them as part of a bigger series, but for now I just would like merging these 2 PRs. The rest still depends on discussions with pypa/wheel and pypa/packaging. ↩

[jaraco]

No problem. Please proceed. I trust your instincts.