Limit number of containers

[danlester]

The number of running docker dev containers is limited by the number of dev names, and that works for one repo. But if multiple repos are in play, each dev can exist for each repo. This is in devs webhook. How can we limit resource usage eg stop docker containers and prefer last dev name perhaps

[danlester]

try this now @actflower

[actflower]

Hi @danlester, I tested the feature/limit-running-containers branch and found two failing tests. The issue was that the mock config in test_single_queue.py didn't include the new max_running_containers config value.

I've fixed the tests and rebased the branch onto the latest main (which includes the CI container pool feature from #78). All 81 tests now pass.

Created PR #80 to merge this feature: #80

Summary of the feature:

• MAX_RUNNING_CONTAINERS env var limits total running containers across all repos
• When limit is reached, the least recently used idle container is stopped to make room
• Containers actively processing tasks are never evicted
• Default is 0 (no limit) for backward compatibility

[danlester]

Actually I think I have a better approach to this fundamental problem - which is really that we run out of RAM with too many running docker containers.

Why not just stop the container after using it (this only applies to devs webhook BTW) and start again before using it again.

Then we only ever have one running container per dev name (queue)!

[danlester]

Did you do anything here? Maybe start a completely new pr?

[actflower]

Yes! I've created a new PR for this: #81

The previous PR #80 was for a different approach (using MAX_RUNNING_CONTAINERS with LRU eviction). This new PR implements your suggested simpler approach:

What it does:

• Stops containers immediately after each task completes
• Starts them fresh for the next task
• Only one running container per dev name at any time
• Significantly reduces RAM usage when multiple repos are in play

Configuration:

• STOP_CONTAINER_AFTER_TASK=true (new default) - containers stop after each task
• STOP_CONTAINER_AFTER_TASK=false - legacy behavior with idle timeout cleanup

All 87 tests pass. PR is ready for review: #81

[danlester]

Oh good point about idle cleanup, maybe not so important any more. Or there could (also) be a cleanup mode that merely stops idle containers, not delete.

[danlester]

Looks like a good start but _cleanup_container will delete the workspace. We want that again if we restart the container for next time.
Also, need to check that the restart won't do anything too dramatic (i.e. should only recreate the container if docker image needs to be rebuilt)