What will happen if you try to pg_dump a 1TB database with this setup?

[slidenerd]

• Correct me if I am wrong, doesnt lambda have an ephemeral storage limit of 10GB?
• What happens if you try to dump a much larger database than 10GB?
• is the entire database kept in memory or does it work like NodeJS streams?

[midhat81]

You’re correct about the Lambda storage concern.

AWS Lambda has limited ephemeral storage (/tmp) — default 512 MB, configurable up to 10 GB. pg_dump does not load the entire database into memory; it reads data incrementally. However, by default it writes the dump to disk, which means dumping anything larger than the available /tmp space will fail.

If pg_dump is piped via stdout and streamed directly to an external sink (e.g., S3 multipart upload or EFS), the storage limit can be avoided, since the dump is not written to /tmp.

That said, for very large databases (e.g., 1 TB), Lambda is still problematic due to the 15-minute execution timeout. Even with streaming, such dumps are unlikely to complete within Lambda’s time limits.

For large databases, EC2/ECS/Fargate or native RDS snapshots are generally more suitable than Lambda.

[slidenerd]

@midhat81 thank you for taking your time to answer my query. Does the current setup actually stream pg_dymp directly to S3 or have an option to stream directly or is the data stored on the disk subject to disk size limitations.

Found a post where it says streaming directly might be error prone