Mature CI

[dimos-bot]

We have a series of interdependant docker images, that need to be rebuilt only sometimes (when relevant files change)

ros -> ros-python -> ros-dev
ubuntu -> python -> dev

In the future we might have more, for different platforms, jetson linux distro etc

On top of these images we run tests.

Requirements:

• We need to rebuild these images sometimes, only if relevant files to that image have changed, and images above always need to be rebuilt if images below have changed.
• As soon as image has built, we can run tests we need to run on that image, don't wait for full docker build.
• Images that are not interdependant but need to be built, need to be built in parallel.
• Once builds have finished, we need to run tests on top of the newly built images. sometimes also in parallel.
  (ros-tests, runs on ros-python mypy, heavy-tests run in parallel on python-dev etc)
• Never build the same image twice, currently if pyproject.toml changes, every subsequent commit to a PR triggers python, ros-python image rebuild, even if image for that change has been built already for prev commit. So images to be built/files changed need to be checked by commit diff, not PR diff from dev.

You can tag images built with something like dimos/dev:pr_number or something, potentially delete after merge?
Cache images, don't pull dev from the internet onto a test runner if dev hasn't changed.

in our .github/worflows/docker.yml we have files for rebuilds defined

      - uses: actions/checkout@v4
      - id: filter
        uses: dorny/paths-filter@v3
        with:
          base: ${{ github.event.before }}
          filters: |
            # ros and python are (alternative) root images
            # change to root stuff like docker.yml etc triggers rebuild of those
            # which cascades into a full rebuild
            ros:
              - .github/workflows/_docker-build-template.yml
              - .github/workflows/docker.yml
              - docker/ros/**

            python:
              - .github/workflows/_docker-build-template.yml
              - .github/workflows/docker.yml
              - docker/python/**
              - pyproject.toml

            dev:
              - docker/dev/**

            tests:
              - dimos/**

Sketch for implementation

(this is just a proposal, whoever hits above targets owns the decisions)

Check our github workflows, the language for what we want is super contrived. and personally I'd write a small python script to orchestrate this,

I'd have just a Action that runs for a PR, main_meta_raw_py_action.py

this action just holds an async dependancy tree, and runs actions using github API until the whole tree is resolved. each node of a tree is a github action it schedules. bonus points if we can dynamically add actions to the PR actions so it's visible in the github UI

(I'm sure there is some asyncio interdependant task runner lib, but super easy to write anyway)

┌───┐┌────────┐                                              
│ros││python  │                                              
└┬──┘└───────┬┘                                              
┌▽─────────┐┌▽──────────────────────────────┐                
│ros-python││dev                            │                
└┬─────────┘└────┬──────┬────────────┬─────┬┘                
┌▽─────────────┐┌▽────┐┌▽──────────┐┌▽───┐┌▽────────────────┐
│ros-python-dev││tests││heavy_tests││mypy││integration_tests│
└┬─────────────┘└─────┘└───────────┘└────┘└─────────────────┘
┌▽───────────────┐                                           
│ros-python-tests│                                           
└────────────────┘

---

Synced from DIM-322