-
Notifications
You must be signed in to change notification settings - Fork 25
Create pgweb app #329
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
pantherman594
wants to merge
17
commits into
master
Choose a base branch
from
jczerk/db_guis
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Create pgweb app #329
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
65ddac9
Add pgweb app.
invalid-email-address e4e8760
Add bookmarks for workspace database instances.
invalid-email-address 0478afc
fixup
invalid-email-address d8fae80
fixup
invalid-email-address 325ca7f
fixup
invalid-email-address b8f27dc
fixup
invalid-email-address 93a2cdf
fixup
invalid-email-address 9ae82e4
fixup
invalid-email-address f3cc88f
fixup
invalid-email-address 1e12811
fixup
invalid-email-address d7f9057
fixup
invalid-email-address ad73cc8
fixup
invalid-email-address 45a75a8
fixup
invalid-email-address cf3067e
doc update
invalid-email-address 1ef5afd
fixup
invalid-email-address ce53145
Clean up shellcheck disables
pantherman594 1032db4
nounset
pantherman594 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| { | ||
| "name": "pgweb", | ||
| "dockerComposeFile": "docker-compose.yaml", | ||
| "service": "app", | ||
| "shutdownAction": "none", | ||
| "workspaceFolder": "/workspace", | ||
| "postCreateCommand": [ | ||
| "./startupscript/post-startup.sh", | ||
| "root", | ||
| "/root", | ||
| "${templateOption:cloud}", | ||
| "${templateOption:login}" | ||
| ], | ||
| "postStartCommand": "/workspace/start-bookmark-refresh.sh", | ||
| "features": { | ||
| "ghcr.io/devcontainers/features/common-utils:2": { | ||
| "installZsh": false, | ||
| "installOhMyZsh": false, | ||
| "upgradePackages": false | ||
| }, | ||
| "ghcr.io/devcontainers/features/java:1": { | ||
| "version": "17" | ||
| }, | ||
| "ghcr.io/devcontainers/features/aws-cli:1": {} | ||
| }, | ||
| "remoteUser": "root" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # pgweb | ||
|
|
||
| Custom Workbench application for querying PostgreSQL databases using pgweb - a lightweight, web-based database browser. | ||
|
|
||
| ## Configuration | ||
|
|
||
| - **Image**: sosedoff/pgweb | ||
| - **Port**: 8081 | ||
| - **User**: root | ||
| - **Home Directory**: /root | ||
| - **Sessions Mode**: Enabled (allows interactive login via web UI) | ||
|
|
||
| ## Access | ||
|
|
||
| Once deployed in Workbench, access the pgweb UI at the app URL (port 8081). | ||
|
|
||
| ## Automatic Database Discovery | ||
|
|
||
| The app automatically discovers all Aurora databases in your Workbench workspace and creates pre-configured connection bookmarks with fresh IAM authentication tokens. | ||
|
|
||
| ### How It Works | ||
|
|
||
| 1. **Auto-Discovery**: Every 10 minutes, the app queries `wb resource list` to find all Aurora databases | ||
| 2. **Access-Based Credentials**: For each database, attempts to get credentials based on your workspace permissions: | ||
| - **Read-Only**: Always attempted first - if successful, creates a read-only bookmark | ||
| - **Write-Read**: Only attempted if you have write access - creates a write-read bookmark if successful | ||
| 3. **IAM Token Generation**: Generates fresh IAM authentication tokens for each access level you have | ||
| 4. **Bookmark Creation**: Creates pgweb bookmarks only for the access levels you're granted | ||
| 5. **Always Fresh**: Tokens refresh every 10 minutes (they expire after 15), so connections never expire | ||
|
|
||
| **Note**: You'll only see bookmarks for databases you have access to. If you only have read-only access to a database, you'll only see the read-only bookmark. If a database is removed from the workspace or your access is revoked, its bookmarks will disappear on the next refresh. | ||
|
|
||
| ### Using Bookmarks | ||
|
|
||
| When you open pgweb, you'll see bookmarks for databases you have access to. Examples: | ||
|
|
||
| - `aurora-demo-db-20260115 (Read-Only)` - Read-only connection | ||
| - `aurora-demo-db-20260115 (Write-Read)` - Read-write connection (only if you have write access) | ||
| - `dc-database (Read-Only)` - Read-only connection to referenced database | ||
| - `dc-database (Write-Read)` - Read-write connection (only if you have write access) | ||
|
|
||
| Click any bookmark to connect instantly - no need to enter credentials! | ||
|
|
||
| ### Manual Connections | ||
|
|
||
| You can also use the interactive login form to enter connection details manually: | ||
|
|
||
| - **Host**: Your Aurora cluster endpoint | ||
| - **Port**: `5432` | ||
| - **Username**: Your database username | ||
| - **Password**: Your database password (works with IAM temporary passwords) | ||
| - **Database**: Your database name | ||
| - **SSL Mode**: `require` | ||
|
|
||
| ## Aurora PostgreSQL with IAM Authentication | ||
|
|
||
| This app is optimized for Aurora PostgreSQL with IAM authentication. The automatic bookmark system handles token refresh transparently, and manual connections support entering temporary IAM passwords directly without URL encoding issues. | ||
|
|
||
| ## Local Testing | ||
|
|
||
| For local testing of the bookmark refresh script: | ||
|
|
||
| ```bash | ||
| # Test with custom paths (useful for local development) | ||
| WB_EXE="$(which wb)" PGWEB_BASE=/tmp/pgweb ./src/pgweb/refresh-bookmarks.sh | ||
| ``` | ||
|
|
||
| Environment variables: | ||
|
|
||
| - `WB_EXE` - Path to wb executable (default: `/usr/bin/wb`) | ||
| - `PGWEB_BASE` - Base directory for pgweb config (default: `/root/.pgweb`) | ||
|
|
||
| For full devcontainer testing: | ||
|
|
||
| 1. Create Docker network: `docker network create app-network` | ||
| 2. Run the app: `devcontainer up --workspace-folder .` | ||
| 3. Access at: `http://localhost:8081` | ||
|
|
||
| ## Customization | ||
|
|
||
| Edit the following files to customize your app: | ||
|
|
||
| - `.devcontainer.json` - Devcontainer configuration and features | ||
| - `docker-compose.yaml` - Docker Compose configuration (change the `command` to customize pgweb options) | ||
| - `devcontainer-template.json` - Template options and metadata | ||
|
|
||
| ## Testing | ||
|
|
||
| To test this app template: | ||
|
|
||
| ```bash | ||
| cd test | ||
| ./test.sh pgweb | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| 1. Fork the repository | ||
| 2. Modify the configuration files as needed | ||
| 3. In Workbench UI, create a custom app pointing to your forked repository | ||
| 4. Select this app template (pgweb) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| { | ||
| "id": "pgweb", | ||
| "version": "1.0.0", | ||
| "name": "pgweb", | ||
| "description": "Web-based PostgreSQL database browser for querying Aurora and other PostgreSQL databases", | ||
| "options": { | ||
| "cloud": { | ||
| "type": "string", | ||
| "enum": ["gcp", "aws"], | ||
| "default": "gcp", | ||
| "description": "Cloud provider (gcp or aws)" | ||
| }, | ||
| "login": { | ||
| "type": "string", | ||
| "description": "Whether to log in to workbench CLI", | ||
| "proposals": ["true", "false"], | ||
| "default": "false" | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| services: | ||
| app: | ||
| # The container name must be "application-server" | ||
| container_name: "application-server" | ||
| # This can be either a pre-existing image or built from a Dockerfile | ||
| image: "sosedoff/pgweb" | ||
| # build: | ||
| # context: . | ||
| # Override the default entrypoint to use our custom script | ||
| entrypoint: [] | ||
| command: ["pgweb", "--sessions", "--bind=0.0.0.0", "--listen=8081", "--bookmarks-dir=/root/.pgweb/bookmarks"] | ||
| user: "root" | ||
| restart: always | ||
| volumes: | ||
| - .:/workspace:cached | ||
| - work:/root/work | ||
| # The port specified here will be forwarded and accessible from the | ||
| # Workbench UI. | ||
| ports: | ||
| - 8081:8081 | ||
| # The service must be connected to the "app-network" Docker network | ||
| networks: | ||
| - app-network | ||
| # SYS_ADMIN and fuse are required to mount workspace resources into the | ||
| # container. | ||
| cap_add: | ||
| - SYS_ADMIN | ||
| devices: | ||
| - /dev/fuse | ||
| security_opt: | ||
| - apparmor:unconfined | ||
|
|
||
| volumes: | ||
| work: | ||
|
|
||
| networks: | ||
| # The Docker network must be named "app-network". This is an external network | ||
| # that is created outside of this docker-compose file. | ||
| app-network: | ||
| external: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,166 @@ | ||
| #!/bin/bash | ||
| set -o errexit | ||
| set -o pipefail | ||
| set -o nounset | ||
|
|
||
| # Allow overriding via environment for local testing | ||
| readonly WB_EXE="${WB_EXE:-/usr/bin/wb}" | ||
| readonly PGWEB_BASE="${PGWEB_BASE:-/root/.pgweb}" | ||
| readonly BOOKMARK_DIR="${PGWEB_BASE}/bookmarks" | ||
|
|
||
| # Create base directory if it doesn't exist | ||
| mkdir -p "${PGWEB_BASE}" | ||
|
|
||
| # Helper function to get credentials and generate IAM auth token | ||
| generate_iam_token() { | ||
| local resource_id="${1}" | ||
| local scope="${2}" | ||
| local endpoint="${3}" | ||
| local port="${4}" | ||
| local username="${5}" | ||
| local region="${6}" | ||
|
|
||
| # Get credentials from Workbench | ||
| local wb_creds | ||
| wb_creds=$(${WB_EXE} resource credentials --id "${resource_id}" --scope "${scope}" --format json 2>/dev/null) || return 1 | ||
| readonly wb_creds | ||
|
|
||
| # Extract AWS credentials | ||
| local access_key secret_key session_token | ||
| access_key=$(echo "${wb_creds}" | jq -r '.AccessKeyId') | ||
| secret_key=$(echo "${wb_creds}" | jq -r '.SecretAccessKey') | ||
| session_token=$(echo "${wb_creds}" | jq -r '.SessionToken') | ||
| readonly access_key secret_key session_token | ||
|
|
||
| # Generate IAM token | ||
| AWS_ACCESS_KEY_ID="${access_key}" \ | ||
| AWS_SECRET_ACCESS_KEY="${secret_key}" \ | ||
| AWS_SESSION_TOKEN="${session_token}" \ | ||
| aws rds generate-db-auth-token \ | ||
| --hostname "${endpoint}" \ | ||
| --port "${port}" \ | ||
| --username "${username}" \ | ||
| --region "${region}" | ||
| } | ||
|
|
||
| # Helper function to create bookmark TOML file | ||
| create_bookmark() { | ||
| local output_file="${1}" | ||
| local endpoint="${2}" | ||
| local port="${3}" | ||
| local username="${4}" | ||
| local password="${5}" | ||
| local database="${6}" | ||
|
|
||
| cat > "${output_file}" <<EOF | ||
| host = "${endpoint}" | ||
| port = ${port} | ||
| user = "${username}" | ||
| password = "${password}" | ||
| database = "${database}" | ||
| sslmode = "require" | ||
| EOF | ||
| } | ||
|
|
||
| refresh_bookmarks() { | ||
| echo "$(date): Refreshing pgweb bookmarks from Workbench resources..." | ||
|
|
||
| # Create temporary directory for new bookmarks (using PID for uniqueness) | ||
| local TEMP_DIR="${PGWEB_BASE}/bookmarks.tmp.$$" | ||
| readonly TEMP_DIR | ||
| rm -rf "${TEMP_DIR}" | ||
| mkdir -p "${TEMP_DIR}" | ||
|
|
||
| # Get list of Aurora databases from Workbench | ||
| local RESOURCES | ||
| RESOURCES=$(${WB_EXE} resource list --format json) | ||
| readonly RESOURCES | ||
|
|
||
| # Process each resource | ||
| echo "${RESOURCES}" | jq -c '.[]' | while read -r resource; do | ||
| local RESOURCE_TYPE | ||
| RESOURCE_TYPE=$(echo "${resource}" | jq -r '.resourceType') | ||
|
|
||
| # Skip non-Aurora resources | ||
| if [[ ! "${RESOURCE_TYPE}" =~ AURORA_DATABASE ]]; then | ||
| continue | ||
| fi | ||
|
|
||
| local RESOURCE_ID | ||
| RESOURCE_ID=$(echo "${resource}" | jq -r '.id') | ||
| echo " Processing: ${RESOURCE_ID} (type: ${RESOURCE_TYPE})" | ||
|
|
||
| # Extract database details from top level (controlled) or referencedResource (reference) | ||
| local DB_DATA | ||
| if [[ "${RESOURCE_TYPE}" == "AWS_AURORA_DATABASE" ]]; then | ||
| DB_DATA="${resource}" | ||
| else | ||
| DB_DATA=$(echo "${resource}" | jq -r '.referencedResource') | ||
| fi | ||
|
|
||
| # Extract database connection info | ||
| local DB_NAME RO_ENDPOINT RO_USER RW_ENDPOINT RW_USER PORT REGION | ||
| DB_NAME=$(echo "${DB_DATA}" | jq -r '.databaseName') | ||
| RO_ENDPOINT=$(echo "${DB_DATA}" | jq -r '.roEndpoint') | ||
| RO_USER=$(echo "${DB_DATA}" | jq -r '.roUser') | ||
| RW_ENDPOINT=$(echo "${DB_DATA}" | jq -r '.rwEndpoint') | ||
| RW_USER=$(echo "${DB_DATA}" | jq -r '.rwUser') | ||
| PORT=$(echo "${DB_DATA}" | jq -r '.port') | ||
| REGION=$(echo "${DB_DATA}" | jq -r '.region // "us-east-1"') | ||
|
|
||
| # Validate all required fields are present | ||
| if [[ -z "${DB_NAME}" || "${DB_NAME}" == "null" ]] || \ | ||
| [[ -z "${RO_ENDPOINT}" || "${RO_ENDPOINT}" == "null" ]] || \ | ||
| [[ -z "${RO_USER}" || "${RO_USER}" == "null" ]] || \ | ||
| [[ -z "${RW_ENDPOINT}" || "${RW_ENDPOINT}" == "null" ]] || \ | ||
| [[ -z "${RW_USER}" || "${RW_USER}" == "null" ]] || \ | ||
| [[ -z "${PORT}" || "${PORT}" == "null" ]]; then | ||
| echo " Missing required database fields, skipping" | ||
| continue | ||
| fi | ||
|
|
||
| # Try to create READ_ONLY bookmark | ||
| echo " Checking read access..." | ||
| local RO_TOKEN | ||
| if RO_TOKEN=$(generate_iam_token "${RESOURCE_ID}" "READ_ONLY" "${RO_ENDPOINT}" "${PORT}" "${RO_USER}" "${REGION}"); then | ||
| echo " Read access confirmed" | ||
| echo " Creating read-only bookmark..." | ||
| create_bookmark "${TEMP_DIR}/${RESOURCE_ID} (Read-Only).toml" "${RO_ENDPOINT}" "${PORT}" "${RO_USER}" "${RO_TOKEN}" "${DB_NAME}" | ||
| echo " Created bookmark: ${RESOURCE_ID} (Read-Only)" | ||
| else | ||
| echo " No read access to ${RESOURCE_ID}, skipping" | ||
| continue | ||
| fi | ||
|
|
||
| # Try to create WRITE_READ bookmark | ||
| echo " Checking write access..." | ||
| local RW_TOKEN | ||
| if RW_TOKEN=$(generate_iam_token "${RESOURCE_ID}" "WRITE_READ" "${RW_ENDPOINT}" "${PORT}" "${RW_USER}" "${REGION}"); then | ||
| echo " Write access confirmed" | ||
| echo " Creating write-read bookmark..." | ||
| create_bookmark "${TEMP_DIR}/${RESOURCE_ID} (Write-Read).toml" "${RW_ENDPOINT}" "${PORT}" "${RW_USER}" "${RW_TOKEN}" "${DB_NAME}" | ||
| echo " Created bookmark: ${RESOURCE_ID} (Write-Read)" | ||
| else | ||
| echo " No write access, skipping write-read bookmark" | ||
| fi | ||
| done | ||
|
|
||
| # Count bookmarks - must use find since the while loop runs in a subshell (due to pipe), | ||
| # so a counter variable incremented in the loop would not be visible here | ||
| local BOOKMARK_COUNT | ||
| BOOKMARK_COUNT=$(find "${TEMP_DIR}" -name "*.toml" -type f 2>/dev/null | wc -l) | ||
| readonly BOOKMARK_COUNT | ||
| echo "$(date): Refresh complete. Created ${BOOKMARK_COUNT} bookmarks." | ||
|
|
||
| # Atomically update symlink to point to new bookmark directory | ||
| ln -sfn "$(basename "${TEMP_DIR}")" "${BOOKMARK_DIR}" | ||
|
|
||
| # Cleanup old bookmark directories (all except current) | ||
| find "${PGWEB_BASE}" -maxdepth 1 -type d -name "bookmarks.tmp.*" ! -name "bookmarks.tmp.$$" -exec rm -rf {} \; | ||
| } | ||
|
|
||
| # Run single refresh | ||
| if ! refresh_bookmarks; then | ||
| echo "$(date): ERROR: Bookmark refresh failed" | ||
| exit 1 | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| #!/bin/bash | ||
| set -o errexit | ||
| set -o pipefail | ||
| set -o nounset | ||
|
|
||
| echo "Starting bookmark refresh for pgweb..." | ||
|
|
||
| # Create base directory (but not bookmarks subdirectory - that will be a symlink) | ||
| mkdir -p /root/.pgweb | ||
|
|
||
| # Make sure refresh script is executable | ||
| chmod +x /workspace/refresh-bookmarks.sh | ||
|
|
||
| # Run initial refresh (blocking) to populate bookmarks before app is marked ready | ||
| echo "Running initial bookmark refresh..." | ||
| /workspace/refresh-bookmarks.sh | ||
|
|
||
| # Start background loop for continuous refresh (detached from parent) | ||
| echo "Starting background bookmark refresh service (every 10 minutes)..." | ||
| # Single quotes intentional: $(date) should expand at runtime, not now | ||
| # shellcheck disable=SC2016 | ||
| nohup bash -c ' | ||
| while true; do | ||
| sleep 600 # 10 minutes | ||
| /workspace/refresh-bookmarks.sh || echo "$(date): WARNING: Bookmark refresh failed" | ||
| done | ||
| ' >> /root/.pgweb/refresh.log 2>&1 & | ||
|
|
||
| REFRESH_PID=$! | ||
| echo "Bookmark refresh service configured (background PID: ${REFRESH_PID})" |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: set -o nounset