Skip to content

feat: add support for max lifetime and idle timeout#407

Open
basanthjenuhb wants to merge 2 commits intoduckdb:mainfrom
basanthjenuhb:feat/connection-lifetime
Open

feat: add support for max lifetime and idle timeout#407
basanthjenuhb wants to merge 2 commits intoduckdb:mainfrom
basanthjenuhb:feat/connection-lifetime

Conversation

@basanthjenuhb
Copy link

@basanthjenuhb basanthjenuhb commented Feb 27, 2026

Add connection lifetime management to PostgresConnectionPool

Problem

The Postgres extension's connection pool keeps connections alive indefinitely once they are opened
https://github.com/duckdblabs/intuit/issues/4
Unlike mature connection pools (e.g. HikariCP), there is no mechanism to:

  • Close connections that have exceeded a maximum age (maxLifetime)
  • Close connections that have been sitting idle in the pool for too long (idleTimeout)

This causes connections to accumulate over time and never be reclaimed, even when workload drops significantly.

Solution

This PR adds two new settings and a background reaper thread to the connection pool:

  • pg_connection_max_lifetime — Maximum age of a pooled connection in seconds since it was first opened. When exceeded, the connection is closed instead of being returned to the cache. Default: 0 (disabled).
  • pg_connection_idle_timeout — Maximum time in seconds a connection can sit idle in the cache before being closed. Default: 0 (disabled).

Both default to 0 to preserve full backward compatibility.

How it works

Connection lifetime is enforced at three points:

  1. On checkout (GetConnectionInternal): When popping a connection from the cache, expired connections (by either max lifetime or idle timeout) are discarded. The pool loops until it finds a valid cached connection or opens a new one.

  2. On return (ReturnConnection): Before caching a connection, the pool checks if it has exceeded max_lifetime. If so, the connection is closed immediately rather than cached. When caching, the returned_at timestamp is set for idle timeout tracking.

  3. Background reaper thread: A dedicated thread periodically scans the cache and removes expired connections. The reaper sleeps for min(max_lifetime, idle_timeout) / 2 between scans (clamped to 1–60 seconds). The reaper is only spawned when at least one setting is non-zero, and is stopped when both are set back to 0.

Each connection now carries a created_at timestamp from when it was first opened. This timestamp flows through PostgresPoolConnection across checkout/return cycles so the pool can accurately track connection age.

Changes

File Description
src/include/storage/postgres_connection_pool.hpp Added CachedConnection struct with timestamps, created_at on PostgresPoolConnection, reaper thread members, SetMaxLifetime/SetIdleTimeout methods
src/storage/postgres_connection_pool.cpp Implemented reaper thread, passive expiry checks on get/return, updated ReturnConnection signature, added destructor for clean shutdown
src/postgres_extension.cpp Registered pg_connection_max_lifetime and pg_connection_idle_timeout extension options with global-scope callbacks
src/storage/postgres_catalog.cpp Read new settings in catalog constructor and apply to pool
test/sql/storage/attach_connection_lifetime.test New test: exercises both settings individually and together, with sleeps to verify expiry and reaping

Usage

-- Close connections older than 30 minutes
SET pg_connection_max_lifetime = 1800;

-- Close connections idle for more than 5 minutes
SET pg_connection_idle_timeout = 300;

-- Disable (default behavior, connections live forever)
SET pg_connection_max_lifetime = 0;
SET pg_connection_idle_timeout = 0;

Test plan

  • New test attach_connection_lifetime.test passes (25 assertions)
  • Existing attach_connection_pool.test passes (no regressions)
  • Verified settings can be enabled/disabled dynamically mid-session
  • Verified queries succeed after connections are reaped (pool creates new ones transparently)
  • Build compiles cleanly on macOS (arm64)

@basanthjenuhb basanthjenuhb changed the title feat: add support for max lifetime and idle timeout feat: add support for max lifetime and idle timeout [WIP - DO NOT REVIEW] Feb 27, 2026
@basanthjenuhb basanthjenuhb changed the title feat: add support for max lifetime and idle timeout [WIP - DO NOT REVIEW] feat: add support for max lifetime and idle timeout Feb 27, 2026
@basanthjenuhb basanthjenuhb marked this pull request as ready for review February 27, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant