pypeline - Define Your CI/CD Pipeline Once, Run It Anywhere

Pypeline lets you define your build, test, and deployment pipeline in a single YAML file and run it consistently across your local development environment and any CI/CD platform (GitHub Actions, Jenkins, etc.). No more platform-specific configurations – write once, run anywhere.

Key Features

• Unified Pipeline Definition: Users can define their entire pipeline in a single YAML file, eliminating the need to switch between different syntaxes and configurations for different CI/CD tools.

• Extensibility: Pypeline supports execution steps defined not only through installed Python packages but also from local scripts.

• Execution Context: Allow sharing information and state between steps. Each step in the pipeline receives an execution context that can be updated during step execution.

• Dependency Handling: Every step can register its dependencies and will only be scheduled if anything has changed.

Installation

Use pipx (or your favorite package manager) to install and run it in an isolated environment:

pipx install pypeline-runner

This will install the pypeline command globally, which you can use to run your pipelines.

Note

The Python package is called pypeline-runner because the name pypeline was already taken on PyPI. The command-line interface is pypeline.

Documentation: pypeline-runner.readthedocs.io

Quick Start

pipx install pypeline-runner
pypeline init --project-dir my-pipeline
cd my-pipeline
pypeline run

Walkthrough: Getting Started with Pypeline

To get started run the init command to create a sample project:

pypeline init --project-dir my-pipeline

The example project pipeline is defined in the pipeline.yaml file.

pipeline:
  - step: CreateVEnv
    module: pypeline.steps.create_venv
    config:
      bootstrap_script: .bootstrap/bootstrap.py
  - step: WestInstall
    module: pypeline.steps.west_install
    description: Download external modules
  - step: MyStep
    file: steps/my_step.py
    description: Run a custom script

This pipeline consists of three steps:

• CreateVEnv: This is a built-in step that creates a Python virtual environment.
• WestInstall: This is a built-in step that downloads external modules using the west tool.
• MyStep: This is a custom step that runs a script defined in the steps/my_step.py file.

You can run the pipeline using the run command:

pypeline run --project-dir my-pipeline

Contributing

The project uses Poetry for dependencies management and packaging. You can set up the development environment using one of the following methods:

Option 1: Using Poetry directly (minimal setup)

poetry install

Option 2: Using pypeline-runner (runs the full pipeline including tests and checks)

pipx install pypeline-runner
pypeline run

To execute the test suite, call pytest inside Poetry's virtual environment via poetry run:

.venv/Scripts/poetry run pytest

For those using VS Code there are tasks defined for the most common commands:

• run tests
• run all checks configured for pre-commit
• generate documentation

See the .vscode/tasks.json for more details.

This repository uses commitlint for checking if the commit message meets the conventional commit format.

Credits

This package was created with Copier and the browniebroke/pypackage-template project template.