Podman with wait strategy `stderr_contains` may be stuck indefinitely

[wiktor-k]

Hi,

We're using rustainers to run integration tests and found out that when running it with high concurrency (e.g. tons of tests via cargo nextest run) the stderr_contains wait strategy makes the container start stuck.

A workaround is using different strategy (e.g. HTTPS) - everything is fine then.

I've made a small repo which reproduces the problem for me:

https://github.com/wiktor-k/nextest-integr-test/blob/e7e25370b75f3698dadef9041c60991688dd0365/src/lib.rs#L51-L54

It seems as if there was some kind of race condition when retrieving the logs but I didn't dig deeper.

CC: @dvzrv

[ilaborie]

Sometime I need to set a lock to avoid multiple tests to run to start the container at same time. I'm using the std::sync::LazyLock for that:

use std::mem;
use std::sync::LazyLock;
use tokio::sync::Mutex;

static LOCK: LazyLock<Mutex<()>> = LazyLock::new(Mutex::default);

const CONTAINER_NAME: &str = "pg_db_test";
const PG_VERSION: &str = "17";

let image = Postgres::default().with_tag(PG_VERSION);
let options = RunOption::builder()
    .with_remove(false)
    .with_name(CONTAINER_NAME)
    .build();

// Need to avoid parallel start of containers
let lock = LOCK.lock().await;
let container = runner.start_with_options(image, options).await?;
mem::drop(lock);

/// ...

Maybe that trick can help in your situation if you really need to use the stderr.

For the issue, it needs a deeper analyze, maybe we need to put that lock mechanism inside the runner?

[wiktor-k]

Yes, the fix seems not to be super-straightforward. I looked at the code for some time but couldn't come up with a solution. Maybe adding a lock would be a good idea. I can definitely help with testing since it's 100% reproducible on my machine.

FWIW once #39 is merged this will be sufficient for me (I'll just use HTTPS check instead of stderr one).

I think it's still good to document the issue, maybe someone else will stumble upon it and will have a starting point.