feat: add support for max lifetime and idle timeout#407
Open
basanthjenuhb wants to merge 2 commits intoduckdb:mainfrom
Open
feat: add support for max lifetime and idle timeout#407basanthjenuhb wants to merge 2 commits intoduckdb:mainfrom
basanthjenuhb wants to merge 2 commits intoduckdb:mainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add connection lifetime management to PostgresConnectionPool
Problem
The Postgres extension's connection pool keeps connections alive indefinitely once they are opened
https://github.com/duckdblabs/intuit/issues/4
Unlike mature connection pools (e.g. HikariCP), there is no mechanism to:
maxLifetime)idleTimeout)This causes connections to accumulate over time and never be reclaimed, even when workload drops significantly.
Solution
This PR adds two new settings and a background reaper thread to the connection pool:
pg_connection_max_lifetime— Maximum age of a pooled connection in seconds since it was first opened. When exceeded, the connection is closed instead of being returned to the cache. Default:0(disabled).pg_connection_idle_timeout— Maximum time in seconds a connection can sit idle in the cache before being closed. Default:0(disabled).Both default to
0to preserve full backward compatibility.How it works
Connection lifetime is enforced at three points:
On checkout (
GetConnectionInternal): When popping a connection from the cache, expired connections (by either max lifetime or idle timeout) are discarded. The pool loops until it finds a valid cached connection or opens a new one.On return (
ReturnConnection): Before caching a connection, the pool checks if it has exceededmax_lifetime. If so, the connection is closed immediately rather than cached. When caching, thereturned_attimestamp is set for idle timeout tracking.Background reaper thread: A dedicated thread periodically scans the cache and removes expired connections. The reaper sleeps for
min(max_lifetime, idle_timeout) / 2between scans (clamped to 1–60 seconds). The reaper is only spawned when at least one setting is non-zero, and is stopped when both are set back to0.Each connection now carries a
created_attimestamp from when it was first opened. This timestamp flows throughPostgresPoolConnectionacross checkout/return cycles so the pool can accurately track connection age.Changes
src/include/storage/postgres_connection_pool.hppCachedConnectionstruct with timestamps,created_atonPostgresPoolConnection, reaper thread members,SetMaxLifetime/SetIdleTimeoutmethodssrc/storage/postgres_connection_pool.cppReturnConnectionsignature, added destructor for clean shutdownsrc/postgres_extension.cpppg_connection_max_lifetimeandpg_connection_idle_timeoutextension options with global-scope callbackssrc/storage/postgres_catalog.cpptest/sql/storage/attach_connection_lifetime.testUsage
Test plan
attach_connection_lifetime.testpasses (25 assertions)attach_connection_pool.testpasses (no regressions)