Remove zombie node logic and add out-of-sync fallback selection by poopoothegorilla · Pull Request #85 · smartcontractkit/chainlink-framework

poopoothegorilla · 2026-02-25T23:40:52Z

Description

Remove "zombie" node logic that incorrectly kept unhealthy nodes alive when they were the last node in the pool, which caused missed state transitions and silent metric gaps (PLEX-2538)
Add fallback node selection that uses out-of-sync nodes when no alive nodes are available, with automatic upgrade back to alive nodes when they recover

Problem

When two nodes (A and B) exist and node A is in FinalizedBlockOutOfSync state, LatestChainInfo() does not count it as alive (since State() returns nodeStateFinalizedBlockOutOfSync). If node B then fails health checks, the l < 2 guard in the alive loop mistakenly treats B as the "last alive node" and keeps it in a zombie state instead of transitioning it to Unreachable or OutOfSync. These zombie nodes are invisible to metrics, preventing proper alerting.

Changes

node_lifecycle.go: Remove all zombie node guards from aliveLoop (4 locations) and outOfSyncLoop (1 location). Nodes now always transition to their correct state regardless of pool size. Remove zombieNodeCheckInterval(), msgCannotDisable, and msgDegradedState.
multi_node.go: Add selectOutOfSyncNode() fallback that picks the best out-of-sync node by highest block number. Update awaitNodeSelection() to fall back to out-of-sync nodes when no alive nodes exist. Update selectNode() to upgrade from an out-of-sync active node to an alive one when available. Update checkLease() to handle nil selector results with out-of-sync fallback.

Tests: Update 5 zombie behavior tests to verify correct state transitions. Add 6 new tests for out-of-sync fallback selection, best-node picking, and alive-node upgrade.

Test plan

[x] All existing multinode tests pass (50/50)
[x] Zombie tests updated to verify nodes transition to correct unhealthy state
[x] New tests cover: fallback to OutOfSync node, fallback to FinalizedBlockOutOfSync node, best-node selection by block height, keeping out-of-sync active when no alive available, upgrading from out-of-sync to alive

github-actions · 2026-02-25T23:41:03Z

👋 poopoothegorilla, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

multinode/multi_node.go

dhaidashenko · 2026-02-26T12:23:14Z

multinode/multi_node.go

+// it will try to upgrade to an alive node when one becomes available.
 func (c *MultiNode[CHAIN_ID, RPC]) selectNode(ctx context.Context) (node Node[CHAIN_ID, RPC], err error) {
 	if c.selectionMode == NodeSelectionModeRandomRPC {
 		return c.awaitNodeSelection(ctx)


We also need to adjust random node selection. Previously, we could abort all connections if the node was not alive. However, we now need to keep the connection to usable non alive node if there are no alive nodes.

Could you clarify what specific adjustment is needed here? With zombie logic retained, the last out-of-sync node is forcibly kept alive, so the existing random selection via awaitNodeSelection should work as before. Are you looking for something beyond what the zombie guards already provide?

multinode/multi_node.go

multinode/multi_node_test.go

dhaidashenko · 2026-02-26T17:55:05Z

multinode/node_lifecycle.go

 			// threshold amount of time, mark it broken
 			lggr.Errorw(fmt.Sprintf("RPC endpoint detected out of sync; no new heads received for %s (last head received was %v)", noNewHeadsTimeoutThreshold, localHighestChainInfo.BlockNumber), "nodeState", n.getCachedState(), "latestReceivedBlockNumber", localHighestChainInfo.BlockNumber, "noNewHeadsTimeoutThreshold", noNewHeadsTimeoutThreshold)
 			if n.poolInfoProvider != nil {
-				// if its the only node and its not a proxy, keep waiting for sync (check LatestChainInfo)


Why remove the comments?

Addressed — restored all removed explanatory comments in aliveLoop (zombie guards for no-new-heads and no-new-finalized-heads sections, including proxy notes and check interval notes) and in outOfSyncLoop (load-balanced RPC note).

dhaidashenko · 2026-02-26T17:56:49Z

multinode/node_lifecycle_test.go

-		tests.AssertLogEventually(t, observedLogs, fmt.Sprintf("RPC endpoint failed to respond to %d consecutive polls", pollFailureThreshold))
-		assert.Equal(t, nodeStateAlive, node.State())
-	})
-	t.Run("with threshold poll failures, we are the last node alive, but is a proxy, transitions to unreachable", func(t *testing.T) {


why remove this tests?

Addressed — restored both zombie behavior tests: 'forcibly keeps it alive' (with poolInfo mock returning 0 live nodes excluding caller) and 'but is a proxy, transitions to unreachable'. Updated mocks to use LatestChainInfo(mock.Anything).

dhaidashenko · 2026-02-26T17:58:04Z

multinode/multi_node_test.go

 		node1.AssertNotCalled(t, "UnsubscribeAllExceptAliveLoop")
 		node2.AssertNotCalled(t, "UnsubscribeAllExceptAliveLoop")
 	})
+	t.Run("RandomRPC falls back to out-of-sync node when no alive nodes available", func(t *testing.T) {


no longer needed. We do not need to fallback to out-of-sync node, since the last out-of-sync node will be kept in "zombie" state.

Addressed — removed this test and all other out-of-sync fallback tests. Zombie state is kept as-is.

dhaidashenko · 2026-02-26T17:58:57Z

multinode/multi_node.go

 		TotalDifficulty: big.NewInt(0),
 	}
 	for _, n := range c.primaryNodes {
+		if n.Name() == callerName {


Only exclude from nLiveNodes calculations. We still want to get full ChainInfo

Addressed — LatestChainInfo now only excludes the caller from nLiveNodes count while still aggregating ChainInfo from all alive nodes including the caller.

dhaidashenko · 2026-02-26T17:59:06Z

multinode/multi_node.go

-// Return highest ChainInfo most recently received by the alive nodes.
+// selectOutOfSyncNode picks the best out-of-sync node by highest block number.
+// Returns nil if no out-of-sync nodes are available.
+func (c *MultiNode[CHAIN_ID, RPC]) selectOutOfSyncNode() Node[CHAIN_ID, RPC] {


No longer needed

Addressed — removed selectOutOfSyncNode() and isUsableState() entirely, along with all fallback logic in selectNode, awaitNodeSelection, and checkLease.

dhaidashenko · 2026-02-26T17:59:19Z

multinode/multi_node.go

 	}

+	// If the current active node is out-of-sync, try to find an alive one first
+	if node != nil && isUsableState(node.State()) {


No longer needed.

Addressed — removed the out-of-sync upgrade block from selectNode. The function now only uses the normal nodeSelector path.

zombie nodes

1d39219

poopoothegorilla requested a review from a team as a code owner February 25, 2026 23:40

product-security-plaid-production bot requested a review from silaslenihan February 25, 2026 23:41

poopoothegorilla requested review from DylanTinianov and dhaidashenko February 25, 2026 23:41

dhaidashenko reviewed Feb 26, 2026

View reviewed changes

poopoothegorilla added 2 commits February 26, 2026 11:30

update

e083737

fix comments

89eeb66

dhaidashenko reviewed Feb 26, 2026

View reviewed changes

update

d030e50

Conversation

poopoothegorilla commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Problem

Changes

Test plan

Uh oh!

github-actions bot commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

poopoothegorilla commented Feb 25, 2026 •

edited

Loading