diff --git a/README.md b/README.md index 50c43467..76ec1a94 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ helm repo add pgdogdev https://helm.pgdog.dev helm install pgdog pgdogdev/pgdog ``` -### Docker +### Try in Docker You can try PgDog quickly using Docker. Install [Docker Compose](https://docs.docker.com/compose/) and run: @@ -33,10 +33,10 @@ You can try PgDog quickly using Docker. Install [Docker Compose](https://docs.do docker-compose up ``` -It will take a few minutes to build PgDog from source and launch the containers. Once started, you can connect to PgDog with psql (or any other PostgreSQL client): +Once started, you can connect to PgDog with psql or any other PostgreSQL client: ``` -PGPASSWORD=postgres psql -h 127.0.0.1 -p 6432 -U postgres gssencmode=disable +PGPASSWORD=postgres psql -h 127.0.0.1 -p 6432 -U postgres ``` The demo comes with 3 shards and 2 sharded tables: @@ -49,59 +49,342 @@ SELECT * FROM users WHERE id = 1; SELECT * FROM payments WHERE user_id = 1; ``` -### Monitoring +## Features -PgDog exposes both the standard PgBouncer-style admin database and an OpenMetrics endpoint. The admin database isn't 100% compatible, -so we recommend you use OpenMetrics for monitoring. Example Datadog configuration and dashboard are [included](examples/datadog). +📘 **[Configuration](https://docs.pgdog.dev/configuration/)** -## Features +All PgDog features are configurable and can be turned on and off. PgDog requires 2 configuration files to operate: -### Load balancer +1. `pgdog.toml`: hosts, sharding configuration, and other settings +2. `users.toml`: usernames and passwords + +### Example -PgDog is an application layer (OSI Level 7) load balancer for PostgreSQL. It understands the Postgres protocol, can proxy multiple replicas (and primary) and distributes transactions evenly between databases. It supports multiple strategies, like round robin, random and least active connections. PgDog can also inspect queries and send `SELECT` queries to replicas, and all others to the primary. This allows to proxy all databases behind a single PgDog deployment. +Most options have reasonable defaults, so a basic configuration for a single user +and database running on the same machine is pretty short: -📘 **[Load balancer](https://docs.pgdog.dev/features/load-balancer)** +**`pgdog.toml`** -#### Healthchecks and failover +```toml +[general] +port = 6432 +default_pool_size = 10 + +[[databases]] +name = "pgdog" +host = "127.0.0.1" +``` + +**`users.toml`** + +```toml +[[users]] +name = "alice" +database = "pgdog" +password = "hunter2" +``` -PgDog maintains a real-time list of healthy hosts. When a database fails a healthcheck, it's removed from the active rotation and queries are re-routed to other replicas. This works like an HTTP load balancer, except it's for your database. +If a database in `pgdog.toml` doesn't have a user in `users.toml`, the connection pool for that database will not be created and users won't be able to connect to it. -Failover maximizes database availability and protects against bad network connections, temporary hardware failures or misconfiguration. +If you'd like to try it out, configure the database and user like so: -📘 **[Healthchecks](https://docs.pgdog.dev/features/healthchecks)** +```sql +CREATE DATABASE pgdog; +CREATE USER pgdog PASSWORD 'pgdog' LOGIN; +``` ### Transaction pooling +📘 **[Transactions](https://docs.pgdog.dev/features/transaction-mode)** + Like PgBouncer, PgDog supports transaction (and session) pooling, allowing -100,000s of clients to use just a few PostgreSQL server connections. +thousands of clients to use just a few PostgreSQL server connections. -📘 **[Transactions](https://docs.pgdog.dev/features/transaction-mode)** +Unlike PgBouncer, PgDog can parse and handle `SET` statements and startup options, ensuring session state is set correctly when sharing server connections between clients with different parameters. -### Sharding +### Load balancer + +📘 **[Load balancer](https://docs.pgdog.dev/features/load-balancer/)** + +PgDog is an application layer (OSI Level 7) load balancer for PostgreSQL. It understands the Postgres protocol, can proxy multiple replicas (and primary) and distributes transactions evenly between databases. The load balancer supports 3 strategies: round robin, random and least active connections. + + +**Example** + +The load balancer is enabled automatically when a database has more than one host: + +```toml +[[databases]] +name = "prod" +host = "10.0.0.1" +role = "primary" + +[[databases]] +name = "prod" +host = "10.0.0.2" +role = "replica" +``` + +#### Health checks + +📘 **[Healthchecks](https://docs.pgdog.dev/features/load-balancer/healthchecks/)** -PgDog is able to handle databases with multiple shards by routing queries automatically to one or more shards. Using the native PostgreSQL parser, PgDog understands queries, extracts sharding keys and determines the best routing strategy. For cross-shard queries, PgDog assembles and transforms results in memory, sending them all to the client as if they are coming from a single database. + +PgDog maintains a real-time list of healthy hosts. When a database fails a health check, it's removed from the active rotation and queries are re-routed to other replicas. This works like an HTTP load balancer, except it's for your database. + +Health checks maximize database availability and protect against bad network connections, temporary hardware failures or misconfiguration. + + +#### Single endpoint + +📘 **[Single endpoint](https://docs.pgdog.dev/features/load-balancer/#single-endpoint)** + + +PgDog uses [`pg_query`](https://github.com/pganalyze/pg_query.rs), which includes the PostgreSQL native parser. By parsing queries, PgDog can detect writes (e.g. `INSERT`, `UPDATE`, `CREATE TABLE`, etc.) and send them to the primary, leaving the replicas to serve reads (`SELECT`). This allows applications to connect to the same PgDog deployment for both reads and writes. + + +##### Transactions + +📘 **[Load balancer & transactions](https://docs.pgdog.dev/features/load-balancer/transactions/)** + + +Transactions can execute multiple statements, so in a primary & replica configuration, PgDog routes them to the primary. Clients can indicate a transaction is read-only, in which case PgDog will send it to a replica: + +```sql +BEGIN READ ONLY; +-- This goes to a replica. +SELECT * FROM users LIMIT 1; +COMMIT; +``` + + +#### Failover + +📘 **[Failover](https://docs.pgdog.dev/features/load-balancer/replication-failover/)** + + +PgDog monitors Postgres replication state and can automatically redirect writes to a different database if a replica is promoted. This doesn't replace tools like Patroni that actually orchestrate failovers. You can use PgDog alongside Patroni (or AWS RDS or other managed Postgres host), to gracefully failover live traffic. + + +**Example** + +To enable failover, set all database `role` attributes to `auto` and enable replication monitoring (`lsn_check_delay` setting): + +```toml +[general] +lsn_check_delay = 0 + +[[databases]] +name = "prod" +host = "10.0.0.1" +role = "auto" + +[[databases]] +name = "prod" +host = "10.0.0.2" +role = "auto" +``` + +### Sharding 📘 **[Sharding](https://docs.pgdog.dev/features/sharding/)** +PgDog is able to manage databases with multiple shards. By using the PostgreSQL parser, PgDog extracts sharding keys and determines the best routing strategy for each query. + +For cross-shard queries, PgDog assembles and transforms results in memory, sending all rows to the client as if they are coming from a single database. + +**Example** + +Configuring multiple hosts for the same database with different shard numbers (`shard` setting) enables sharding: + +```toml +[[databases]] +name = "prod" +host = "10.0.0.1" +shard = 0 + +[[databases]] +name = "prod" +host = "10.0.0.2" +shard = 1 +``` + +Note: read below for how to configure query routing. At least one sharded table is required for sharding to work as expected. + +#### Sharding functions + +📘 **[Sharding functions](https://docs.pgdog.dev/features/sharding/sharding-functions/)** + +PgDog has two main sharding algorithms: + +1. PostgreSQL partition functions (`HASH`, `LIST`, `RANGE`) +2. Using schemas + +##### Partition-based sharding + +Partition-based sharding functions are taken directly from Postgres source code. This choice intentionally allows to shard data both with PgDog and with Postgres [foreign tables](https://www.postgresql.org/docs/current/sql-createforeigntable.html) and [`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html). + +**Examples** + +The `PARTITION BY HASH` algorithm is used by default when configuring sharded tables: + +```toml +[[sharded_tables]] +database = "prod" +column = "user_id" +``` + +List-based sharding (same as `PARTITION BY LIST` in Postgres) can be configured as follows: + +```toml +# Sharded table definition still required. +[[sharded_tables]] +database = "prod" +column = "user_id" + +# Value-specific shard mappings. +[[sharded_mapping]] +database = "prod" +column = "user_id" +values = [1, 2, 3, 4] +shard = 0 + +[[sharded_mapping]] +database = "prod" +column = "user_id" +values = [5, 6, 7, 8] +shard = 1 +``` + +For range-based sharding, replace the `values` setting with a range, for example: + +```toml +start = 0 # include +end = 5 # exclusive +``` + +##### Schema-based sharding + +📘 **[Schema-based sharding](https://docs.pgdog.dev/configuration/pgdog.toml/sharded_schemas/)** + +Schema-based sharding works on the basis of PostgreSQL schemas. Tables under the same schema are placed on the same shard and all queries that refer to those tables are routed to that shard automatically. + +**Example** + +Configuring sharded schemas uses a different configuration from sharded tables: + +```toml +[[sharded_schemas]] +database = "prod" +name = "customer_a" +shard = 0 + +[[sharded_schemas]] +database = "prod" +name = "customer_b" +shard = 1 +``` + +Queries that refer tables in schema `customer_a` will be sent to shard 0. For example, a query that refers to a table by its fully-qualified name will be sent to one shard only: + +```sql +INSERT INTO customer_a.orders (id, user_id, amount) +VALUES ($1, $2, $3); +``` + +Alternatively, the schema name can be specified in the `search_path` session variable: + +```sql +SET search_path TO public, customer_a; +-- All subsequent queries will be sent to shard 0. +SELECT * FROM orders LIMIT 1; +``` + +You can also set the `search_path` for the duration of a single transaction, using `SET LOCAL`, ensuring only that transaction is sent to the desired shard: + +```sql +-- The entire transaction will be sent to shard 1. +BEGIN; +SET LOCAL search_path TO public, customer_b; +SELECT * FROM orders LIMIT 1; +COMMIT; +``` + +#### Direct-to-shard queries + +📘 **[Direct-to-shard queries](https://docs.pgdog.dev/features/sharding/query-routing/)** + + +Queries that contain a sharding key are sent to one database only. This is the best case scenario for sharded databases, since the load is uniformly distributed across the cluster. + +**Example**: + +```sql +-- user_id is the sharding key. +SELECT * FROM users WHERE user_id = $1; +``` + +#### Cross-shard queries + +- 📘 **[Cross-shard queries](https://docs.pgdog.dev/features/sharding/cross-shard-queries/)** +- 📘 **[SELECT](https://docs.pgdog.dev/features/sharding/cross-shard-queries/select/)** +- 📘 **[INSERT](https://docs.pgdog.dev/features/sharding/cross-shard-queries/insert/)** +- 📘 **[UPDATE and DELETE](https://docs.pgdog.dev/features/sharding/cross-shard-queries/update/)** +- 📘 **[DDL](https://docs.pgdog.dev/features/sharding/cross-shard-queries/ddl/)** + +Queries with multiple sharding keys or without one are sent to all databases and results are post-processed and assembled in memory. PgDog then sends the final result to the client. + +Currently, support for certain SQL features in cross-shard queries is limited. However, the list of supported ones keeps growing: + +| Feature | Supported | Notes | +|-|-|-| +| Aggregates | Partial | `count`, `min`, `max`, `stddev`, `variance`, `sum`, `avg` are supported. | +| `ORDER BY` | Partial | Column in `ORDER BY` clause must be present in the result set. | +| `GROUP BY` | Partial | Same as `ORDER BY`, referenced columns must be present in result set. | +| Multi-tuple `INSERT` | Supported | PgDog generates one statement per tuple and executes them automatically. | +| Sharding key `UPDATE` | Supported | PgDog generates a `SELECT`, `INSERT` and `DELETE` statements and execute them automatically. | +| Subqueries | No | The same subquery is executed on all shards. | +| CTEs | No | The same CTE is executed on all shards. | + + #### Using `COPY` -PgDog ships with a text/CSV parser and can split `COPY` commands between all shards automatically. This allows clients to ingest data into sharded PostgreSQL without preprocessing. +📘 **[Copy](https://docs.pgdog.dev/features/sharding/cross-shard-queries/copy/)** -📘 **[Copy](https://docs.pgdog.dev/features/sharding/copy/)** +PgDog has a text, CSV & binary parser and can split rows sent via `COPY` command between all shards automatically. This allows clients to ingest data into sharded PostgreSQL without preprocessing. -#### Re-sharding -PgDog understands the PostgreSQL logical replication protocol and can split data between databases in the background and without downtime. This allows to shard existing databases and add more shards to existing clusters in production, without impacting database operations. +#### Consistency (two-phase commit) -📘 **[Re-sharding](https://docs.pgdog.dev/features/sharding/resharding/)** +📘 **[Two-phase commit](https://docs.pgdog.dev/features/sharding/2pc/)** -### Configuration +To make sure cross-shard writes are atomic, PgDog supports Postgres' [two-phase transactions](https://www.postgresql.org/docs/current/two-phase.html). When enabled, PgDog handles `COMMIT` statements sent by clients by executing the 2pc exchange on their behalf: -PgDog is highly configurable and most aspects of its operation can be tweaked at runtime, without having -to restart the process or break connections. If you've used PgBouncer (or PgCat) before, the options -will be familiar. If not, they are documented with examples. +```sql +PREPARE TRANSACTION '__pgdog_unique_id'; +COMMIT PREPARED '__pgdog_unique_id'; +``` + +In case the client disconnects or Postgres crashes, PgDog will automatically rollback the transaction if it's in phase I and commit it if it's in phase II. + +#### Unique identifiers + +📘 **[Unique IDs](https://docs.pgdog.dev/features/sharding/unique-ids/)** + +While applications can use `UUID` (v4 and now v7) to generate unique primary keys, PgDog supports creating unique `BIGINT` identifiers, without using a sequence: + +```sql +SELECT pgdog.unique_id(); +``` + +This uses a timestamp-based algorithm, can produce millions of unique numbers per second and doesn't require an expensive cross-shard index to guarantee uniqueness. + +#### Re-sharding + +📘 **[Re-sharding](https://docs.pgdog.dev/features/sharding/resharding/)** + +PgDog understands the PostgreSQL logical replication protocol and can orchestrate data splits between databases, in the background and without downtime. This allows to shard existing databases and add more shards to existing clusters in production, without impacting database operations. -📘 **[Configuration](https://docs.pgdog.dev/configuration/)** ## Running PgDog locally @@ -122,8 +405,6 @@ PgDog has two configuration files: * `pgdog.toml` which contains general settings and PostgreSQL servers information * `users.toml` for users and passwords -Most options have reasonable defaults, so a basic configuration for a single user -and database running on the same machine is pretty short: **`pgdog.toml`** @@ -142,12 +423,10 @@ password = "pgdog" database = "pgdog" ``` -If you'd like to try this out, you can set it up like so: +### Monitoring -```postgresql -CREATE DATABASE pgdog; -CREATE USER pgdog PASSWORD 'pgdog' LOGIN; -``` +PgDog exposes both the standard PgBouncer-style admin database and an OpenMetrics endpoint. The admin database isn't 100% compatible, +so we recommend you use OpenMetrics for monitoring. Example Datadog configuration and dashboard are [included](examples/datadog). #### Try sharding @@ -167,9 +446,13 @@ name = "pgdog_sharded" host = "127.0.0.1" database_name = "shard_1" shard = 1 + +[[sharded_tables]] +database = "pgdog_sharded" +column = "user_id" ``` -Don't forget to specify a user: +Don't forget to configure a user: **`users.toml`** @@ -182,7 +465,7 @@ password = "pgdog" And finally, to make it work locally, create the required databases: -```postgresql +```sql CREATE DATABASE shard_0; CREATE DATABASE shard_1; @@ -215,19 +498,19 @@ cargo run --release -- -d postgres://user:pass@localhost:5432/db1 -d postgres:// You can connect to PgDog with `psql` or any other PostgreSQL client: ```bash -psql "postgres://pgdog:pgdog@127.0.0.1:6432/pgdog?gssencmode=disable" +psql postgres://pgdog:pgdog@127.0.0.1:6432/pgdog ``` ## 🚦 Status 🚦 -PgDog is used in production and at scale. Most features are stable, while some are experimental. Check [documentation](https://docs.pgdog.dev/features/) for more details. +PgDog is used in production and at scale. Most features are stable, while some are experimental. Check [documentation](https://docs.pgdog.dev/features/) for more details. New sharding features are added almost weekly. ## Performance -PgDog does its best to minimize its impact on overall database performance. Using Rust and Tokio is a great start for a fast network proxy, but additional care is also taken to perform as few operations as possible while moving data between client and server sockets. Some benchmarks are provided to help set a baseline. - 📘 **[Architecture & benchmarks](https://docs.pgdog.dev/architecture/)** +PgDog is heavily optimized for performance. We use Rust, [Tokio](https://tokio.rs/), [bytes crate](https://docs.rs/bytes/latest/bytes/) to avoid unnecessary memory allocations, and profile for performance regressions on a regular basis. + ## License PgDog is free and open source software, licensed under the AGPL v3. While often misunderstood, this license is very permissive diff --git a/pgdog/tests/vector/README.md b/pgdog/tests/vector/README.md index 621e4b0c..37e452f2 100644 --- a/pgdog/tests/vector/README.md +++ b/pgdog/tests/vector/README.md @@ -6,7 +6,7 @@ This demo uses [Cohere/wikipedia](https://huggingface.co/datasets/Cohere/wikiped Install [pgvector](https://github.com/pgvector/pgvector) into your shards. Make sure to run: -```postgresql +```sql CREATE EXTENSION vector; ```