refactor: Consolidate dependencies into pyproject.toml (PEP 621 single source of truth)

[blalterman]

Problem Statement

SolarWindPy currently maintains 5 dependency specification files with significant duplication:

1. pyproject.toml - Package metadata with incomplete optional-dependencies
2. requirements-dev.txt - De facto source of truth (27 packages, triggers automated sync)
3. requirements.txt - Frozen pins generated via pip freeze (for ReadTheDocs)
4. docs/requirements.txt - Generated subset (7 packages) for documentation builds
5. solarwindpy.yml - Conda environment file (generated from requirements-dev.txt)

Maintenance Burden

Every dependency update triggers the sync-requirements.yml workflow which:

• Regenerates requirements.txt via freeze_requirements.py
• Regenerates docs/requirements.txt via generate_docs_requirements.py
• Regenerates solarwindpy.yml via requirements_to_conda_env.py
• Creates automated PRs with these changes

Recent commit history shows this churn:

cd051cb chore: auto-sync requirements from requirements-dev.txt
f4be657 chore: auto-sync requirements from requirements-dev.txt
5e9222c chore: auto-sync requirements from requirements-dev.txt

Core Issue: requirements-dev.txt acts as the source of truth, not pyproject.toml (which violates PEP 621).

---

Proposed Solution

Follow PEP 621 standard and modern Python packaging best practices:

Make pyproject.toml the Single Source of Truth

Consolidate all dependencies into [project.optional-dependencies] following the Astropy pattern (de facto standard for scientific Python):

[project.optional-dependencies]
# Testing infrastructure
test = [
    "pytest>=7.4.4",
    "pytest-cov>=4.1.0",
]

# Documentation building
docs = [
    "sphinx",
    "sphinx_rtd_theme",
    "sphinxcontrib-spelling",
    "sphinxcontrib-bibtex",
    "doc8",
    "numpydoc",
    "docstring-inheritance>=2.0",
]

# Code quality tools
dev = [
    "black",
    "flake8",
    "flake8-docstrings",
    "pydocstyle",
]

# Optional HDF5 support
hdf5 = [
    "tables",  # PyTables
]

# Development tools (not for PyPI distribution)
tools = [
    "gh",  # GitHub CLI
    "psutil>=5.9.0",
]

# Complete development environment
all = [
    "solarwindpy[test,docs,dev,hdf5,tools]",
]

Rationale: Granular groups allow flexible installation patterns:

• CI testing: pip install -e .[test,dev]
• Documentation: pip install -e .[docs]
• Complete dev: pip install -e .[all]

---

Implementation Plan

Phase 1: Restructure pyproject.toml

Action: Add complete [project.optional-dependencies] structure as shown above.

Benefit: Establishes true single source of truth per PEP 621.

---

Phase 2: Update Scripts

Modify scripts/requirements_to_conda_env.py

Current: Reads from requirements-dev.txt

New: Read from pyproject.toml

import tomllib  # Python 3.11+

with open("pyproject.toml", "rb") as f:
    data = tomllib.load(f)

core_deps = data["project"]["dependencies"]
optional = data["project"]["optional-dependencies"]

# Combine groups (exclude 'tools' - gh not on conda-forge)
all_deps = (
    core_deps
    + optional["test"]
    + optional["docs"]
    + optional["dev"]
    + optional["hdf5"]
)

# Continue with existing conda name translation logic...

Update or Remove scripts/freeze_requirements.py

Decision needed: Keep frozen requirements.txt for ReadTheDocs stability?

Option A (recommended): Remove entirely, use live versions from pyproject.toml
Option B: Keep but read from pyproject.toml instead of requirements-dev.txt

---

Phase 3: Simplify ReadTheDocs Configuration

Update .readthedocs.yaml

Current (uses 2 requirements files):

python:
  install:
    - requirements: requirements.txt
    - requirements: docs/requirements.txt
    - method: pip
      path: .

New (reads directly from pyproject.toml):

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs  # Reads [project.optional-dependencies] docs group

Benefit: Eliminates 2 generated requirements files, uses source of truth directly.

Reference: https://docs.readthedocs.io/en/stable/config-file/v2.html#packages

---

Phase 4: Update CI/CD Workflows

.github/workflows/continuous-integration.yml

Change:

# Before
- name: Install dependencies
  run: pip install -r requirements-dev.txt

# After
- name: Install dependencies
  run: pip install -e .[test,dev]

.github/workflows/sync-requirements.yml

Major Simplification:

Before (generates 3 files):

- run: pip install -r requirements-dev.txt
- run: python scripts/generate_docs_requirements.py
- run: python scripts/freeze_requirements.py
- run: python scripts/requirements_to_conda_env.py --overwrite

After (generates 1 file):

- run: pip install -e .[all]
- run: python scripts/requirements_to_conda_env.py --overwrite

Other Workflows

Update any workflow using requirements-dev.txt:

• .github/workflows/publish.yml → pip install -e .[dev,test]
• Check .github/workflows/release-pipeline.yml for usage

---

Phase 5: Remove Obsolete Files

Delete:

• ❌ requirements-dev.txt - Replaced by pyproject.toml [project.optional-dependencies]
• ❌ docs/requirements.txt - ReadTheDocs uses pyproject.toml directly via extra_requirements
• ❌ scripts/generate_docs_requirements.py - No longer needed

Keep (generated from pyproject.toml):

• ✅ solarwindpy.yml - Conda users need this for environment creation
• ⚠️ requirements.txt - Decision: Remove or keep for docs stability?

---

Phase 6: Update Documentation

README.rst

Update development installation section:

Development Installation
------------------------

For complete development environment::

    pip install -e .[all]

Or install specific groups::

    pip install -e .[test]       # Testing only
    pip install -e .[docs]       # Documentation building
    pip install -e .[dev]        # Code quality tools
    pip install -e .[hdf5]       # Optional HDF5 support

docs/source/installation.rst

Update corresponding development setup documentation.

---

Expected Benefits

Before (Current State)

• 5 dependency files (2 manual, 3 generated)
• requirements-dev.txt is de facto source of truth ❌
• Automated sync creates PR churn
• Duplication across files increases maintenance burden

After (Proposed State)

• 1 source file: pyproject.toml (PEP 621 standard) ✅
• 1 generated file: solarwindpy.yml (for conda users)
• Simpler CI workflows
• Follows scientific Python best practices (Astropy pattern)
• Eliminates generate_docs_requirements.py script
• Standard pip install -e .[groups] pattern

---

Decision Points

Q1: Frozen requirements.txt for ReadTheDocs?

Option A (recommended): Remove entirely, always use latest compatible versions

• Pros: Simpler, catches incompatibilities early
• Cons: Docs builds might break on upstream updates

Option B: Keep but generate from pyproject.toml

• Pros: Stable docs builds
• Cons: Additional maintenance

Recommendation: Start with Option A, only revert if docs builds become unstable.

Q2: When to Execute?

Option A: Immediately (before v0.2.0 release)
Option B: Bundle with next feature work
Option C: Make it part of v0.2.0 as infrastructure improvement

Q3: Investigate pyproject2conda?

Option A (current): Keep custom requirements_to_conda_env.py

• Pros: Simple, works well, we control it
• Cons: Manual maintenance of pip→conda name translations

Option B: Migrate to pyproject2conda

• Pros: Industry standard, more features, community maintained
• Cons: Learning curve, requires [tool.pyproject2conda] configuration

Recommendation: Keep custom script (simpler), revisit pyproject2conda if complexity grows.

---

Files Modified

Source Files

• pyproject.toml - Add complete [project.optional-dependencies]
• scripts/requirements_to_conda_env.py - Read from pyproject.toml
• scripts/freeze_requirements.py - Update or remove (decision needed)
• .readthedocs.yaml - Use extra_requirements method
• .github/workflows/continuous-integration.yml - Replace requirements-dev.txt
• .github/workflows/sync-requirements.yml - Simplify workflow
• .github/workflows/publish.yml - Replace requirements-dev.txt
• README.rst - Update installation instructions
• docs/source/installation.rst - Update development setup

Files Deleted

• requirements-dev.txt
• docs/requirements.txt
• scripts/generate_docs_requirements.py

Files Generated (Post-Commit)

• solarwindpy.yml - Auto-generated by updated script
• requirements.txt - Optional, if frozen pins desired

---

Implementation Checklist

• Create feature branch: refactor/single-source-dependencies
• Update pyproject.toml with complete optional-dependencies
• Modify scripts/requirements_to_conda_env.py to read from pyproject.toml
• Test conda generation: python scripts/requirements_to_conda_env.py --overwrite
• Update .readthedocs.yaml to use extra_requirements
• Update all CI workflows to use pip install -e .[groups]
• Test locally: pip install -e .[all] + run tests
• Remove obsolete files: git rm requirements-dev.txt docs/requirements.txt scripts/generate_docs_requirements.py
• Update README.rst and docs/source/installation.rst
• Commit with conventional commit message
• Push and monitor CI
• Monitor ReadTheDocs build success
• Update this issue with results

---

References

Standards

• PEP 621: Storing project metadata in pyproject.toml
• PyPA pyproject.toml Specification
• Python Packaging User Guide: Writing pyproject.toml

Tools

• ReadTheDocs Configuration Reference
• pyproject2conda Documentation
• Grayskull (conda recipe generator)

Examples

• Astropy pyproject.toml - Excellent example for scientific Python
• NumPy pyproject.toml - Complex build requirements

---

Related Issues

• Complements the Python 3.11 minimum version bump (#issue-from-feat-branch)
• Prepares infrastructure for v0.2.0 release
• Addresses technical debt in dependency management

---

Priority: Medium
Type: Refactoring / Infrastructure Improvement
Breaking Change: No (transparent to users)
Effort: ~4-6 hours (implementation + testing + docs)