Skip to content

[flagd] Improve wait in init #1675

@toddbaert

Description

@toddbaert

The flagd init code is not ideal.

  • the FlagdProvider's initialize function is called, which calls flagResolver.init()... the method then blocks waiting for syncResources.waitForInitialization(...)
  • syncResources.waitForInitialization(...) always runs until the deadline timeout; it would be ideal if it simply immediately returned if an error occurs, and only timed-out if nothing happens
  • the error returned by initialize does not return the exception thrown by the stream

Fixing the above should improve the DX of the provider, and ease troubleshooting.

More background from a previous PR is below:

@leakonvalinka ... I kept seeing intermittent test failures with this PR (this was the biggest example. At first, I thought it was a race-condition in the test itself, but it was not). I have removed this now, because I found the root cause:

The main issue was here. It was not any logical error with the code you added, but your changes surfaced an existing issue. The race condition works like this:

  • the FlagdProvider's initialize function is called, which calls flagResolver.init()... the method then blocks waiting for syncResources.waitForInitialization(...)
  • normally, in error situations, this method then times out (another thread runs error handlers)
  • however, in our situation, this method doesn't just time out; due to our changes to shut down when FATAL errors are seen, the method starting throwing IllegalStateException (see the condition at the end of the method)
  • since this is not a FATAL error, this was thrown by init, and depending on timing, would "override" the FATAL with a non-fatal error (the IllegalStateException)

My fix to throw the Fatal error depending on the last event is a bandaid fix, because this PR has already been open for a while.

To really fix the issue, we should improve the way the waiting in the init works, and actually capture the error during init, and not wait for the full timeout every time. I will open a new issue for this.

cc @guidobrei @aepfli

Originally posted by @toddbaert in #1624 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions