Skip to content

Gateway Remote Debug #1233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
renuka-fernando wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Gateway Remote Debug #1233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ef406c0
Make control plane host optional
renuka-fernando Feb 18, 2026
9f7e1b5
build(gateway): remove build-local targets from all gateway Makefiles
renuka-fernando Feb 18, 2026
ae3b074
Build gateway remote debug images
renuka-fernando Feb 18, 2026
65826aa
Update gateway debug guide with remote debug
renuka-fernando Feb 18, 2026
305af97
Add a note to sync
renuka-fernando Feb 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .vscode/launch.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,5 +83,37 @@
"APIP_GW_ANALYTICS_ACCESS__LOGS__SERVICE_MODE": "tcp",
},
},
{
"name": "Gateway Controller (Remote)",
"type": "go",
"request": "attach",
"mode": "remote",
"port": 2345,
"host": "127.0.0.1",
"apiVersion": 2,
"trace": "log",
"substitutePath": [
{
"from": "${workspaceFolder}/gateway/gateway-controller",
"to": "/build"
}
]
},
{
"name": "Policy Engine (Remote)",
"type": "go",
"request": "attach",
"mode": "remote",
"port": 2346,
"host": "127.0.0.1",
"apiVersion": 2,
"trace": "log",
"substitutePath": [
{
"from": "${workspaceFolder}/gateway/gateway-runtime/policy-engine",
"to": "/api-platform/gateway/gateway-runtime/policy-engine"
}
]
},
]
}
113 changes: 97 additions & 16 deletions gateway/DEBUG_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,80 @@
# Gateway Local Debug Guide
# Gateway Debug Guide

This guide explains how to debug the gateway with **Gateway Controller** and **Policy Engine** running locally in VS Code, and the **Gateway Runtime (Router)** running in Docker Compose.
Two debug options are available. **Option 1 (Remote Debug)** is the recommended approach — everything runs in Docker and VS Code attaches via dlv. **Option 2 (Local Process)** runs the controller and policy engine as local VS Code processes with only the router in Docker.

## Architecture Overview
---

## Option 1 (Recommended): Remote Debug — All Components in Docker

Gateway Controller and Policy Engine run inside Docker containers with dlv in server mode. VS Code attaches remotely.

### Step 1: Build Debug Images

```bash
cd gateway
make build-debug
```

This builds both `gateway-controller-debug:latest` and `gateway-runtime-debug:latest`.

### Step 2: Start the Full Stack

```bash
cd gateway
docker compose -f docker-compose.debug.yaml up
```
Comment on lines +22 to +25
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Usage command here conflicts with the header comment in docker-compose.debug.yaml.

The guide correctly runs the debug compose file standalone (-f docker-compose.debug.yaml), but docker-compose.debug.yaml line 20 says:

# Usage: docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up

Since the debug file defines complete service definitions (not partial overrides), running it as an overlay on top of docker-compose.yaml would cause port-binding conflicts. One of the two should be updated for consistency — the guide appears correct here, so the comment in docker-compose.debug.yaml should be fixed.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@gateway/DEBUG_GUIDE.md` around lines 22 - 25, The header comment inside
docker-compose.debug.yaml currently suggests using it as an overlay ("Usage:
docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up"), which
conflicts with the README command in DEBUG_GUIDE.md that runs
docker-compose.debug.yaml standalone; update the comment in
docker-compose.debug.yaml (the "Usage: docker compose -f docker-compose.yaml -f
docker-compose.debug.yaml up" line) to reflect standalone usage (e.g., instruct
to run "docker compose -f docker-compose.debug.yaml up") or otherwise clarify
that this file contains full service definitions and must not be used as an
override, so the two docs are consistent.


Wait until you see both containers are ready. The policy engine waits up to 30 seconds for dlv startup before the socket becomes available.

### Step 3: Set Breakpoints

Open the relevant source files in VS Code and set breakpoints:

- **Gateway Controller**: files under `gateway/gateway-controller/`
- **Policy Engine**: files under `gateway/gateway-runtime/policy-engine/`

### Step 4: Attach VS Code Debugger

In the VS Code **Run & Debug** panel, launch:

- **"Gateway Controller (Remote)"** — attaches to `localhost:2345`
- **"Policy Engine (Remote)"** — attaches to `localhost:2346`

Both can be attached simultaneously. Source path substitution is configured automatically in `.vscode/launch.json`:

| Component | Local path | Container path |
|---|---|---|
| Gateway Controller | `gateway/gateway-controller` | `/build` |
| Policy Engine | `gateway/gateway-runtime/policy-engine` | `/api-platform/gateway/gateway-runtime/policy-engine` |

### Step 5: Deploy an API and Trigger Breakpoints

```bash
# Deploy a test API
curl -X POST http://localhost:9090/apis \
-H "Content-Type: application/yaml" \
--data-binary @path/to/api.yaml

# Send a request through the router
curl http://localhost:8080/petstore/v1/pets
```

### Notes

- dlv runs with `--accept-multiclient` — you can detach and re-attach without restarting containers.
- Containers run as root (required by dlv for ptrace); resource limits are removed for debug headroom.
- Policy Engine socket wait timeout is 30s (vs 10s in production) to account for dlv startup overhead.
- All ports remain accessible: `9090` (Controller REST), `8080`/`8443` (Router), `9002` (PE admin), `18000`/`18001` (xDS).

---

## Option 2 (Alternative): Local Process Debug — Controller + Policy Engine in VS Code

Gateway Controller and Policy Engine run as local VS Code processes. Only the Envoy Router runs in Docker Compose.

> **Warning:** Processes run directly on the host, so Go resolves modules via `go.work`. Local versions of `sdk` and other workspace modules are used instead of the published Go module versions — including any uncommitted or untagged changes. Behavior may differ from a production build.

### Architecture

```mermaid
graph TB
Expand All @@ -21,14 +93,12 @@ graph TB
GC -->|localhost:18001| PE
```

## Prerequisites
### Prerequisites

- VS Code with Go extension installed
- Docker and Docker Compose
- Control plane host and registration token (optional, for gateway registration)

## Step-by-Step Instructions

### Step 1: Configure Control Plane Connection

Update `.vscode/launch.json` in the **Gateway Controller** configuration with your control plane details:
Expand All @@ -48,22 +118,33 @@ Update `.vscode/launch.json` in the **Gateway Controller** configuration with yo

### Step 2: Update Docker Compose Configuration

Edit `gateway/docker-compose.yml` and comment out the policy engine port mappings in the `gateway-runtime` service, keeping only the router ports:
In `gateway/docker-compose.yaml`, make two changes to the `gateway-runtime` service:

1. Set `GATEWAY_CONTROLLER_HOST` to `host.docker.internal` so the runtime reaches the locally-running controller:

```yaml
services:
gateway-runtime:
environment:
GATEWAY_CONTROLLER_HOST: host.docker.internal
```

2. Comment out the **Policy Engine** port block:

```yaml
services:
gateway-runtime:
ports:
# Router ports (keep these)
- "8080:8080" # HTTP
- "8443:8443" # HTTPS
- "9901:9901" # Admin

# Policy engine ports (comment these out - running locally)
# - "9001:9001" # ext_proc
# - "9002:9002" # Policy engine admin
# Router (Envoy) - keep these
- "8080:8080" # HTTP ingress
- "8443:8443" # HTTPS ingress
- "8081:8081" # xDS-managed API listener
- "8082:8082" # WebSub Hub dynamic forward proxy
- "8083:8083" # WebSub Hub internal listener
- "9901:9901" # Envoy admin
# Policy Engine - comment these out
# - "9002:9002" # Admin API
# - "9003:9003" # Metrics
# - "18090:18090" # ALS
```

### Step 3: Start Gateway Controller
Expand Down
32 changes: 16 additions & 16 deletions gateway/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,12 @@ help: ## Show this help message
@echo ''
@echo 'Build Targets:'
@echo ' make build - Build all gateway component Docker images (buildx)'
@echo ' make build-local - Build all gateway component Docker images locally (faster)'
@echo ' make build-controller - Build gateway-controller Docker image'
@echo ' make build-controller-debug - Build debug gateway-controller image for remote debugging (dlv on port 2345)'
@echo ' make build-gateway-builder - Build gateway-builder Docker image'
@echo ' make build-gateway-runtime - Build gateway-runtime Docker image (Router + Policy Engine)'
@echo ' make build-local-gateway-runtime - Build gateway-runtime Docker image locally (faster)'
@echo ' make build-debug - Build all debug images (controller + runtime) for remote debugging'
@echo ' make build-gateway-runtime-debug - Build debug gateway-runtime image for remote debugging (dlv on port 2346)'
@echo ''
@echo 'Multi-arch Build and Push Targets:'
@echo ' make build-and-push-multiarch - Build and push all gateway components for multiple architectures'
Expand Down Expand Up @@ -79,8 +80,9 @@ help: ## Show this help message
.PHONY: build
build: build-controller build-gateway-builder build-gateway-runtime ## Build all gateway components

.PHONY: build-local
build-local: build-local-controller build-local-gateway-builder build-local-gateway-runtime ## Build all gateway components locally (faster)

.PHONY: build-debug
build-debug: build-controller-debug build-gateway-runtime-debug ## Build all debug images (controller + runtime) for remote debugging

.PHONY: build-coverage
build-coverage: build-gateway-builder build-controller-coverage build-gateway-runtime-coverage ## Build all coverage-instrumented gateway components
Expand All @@ -90,41 +92,39 @@ build-controller: ## Build gateway-controller Docker image
@echo "Building controller Docker image ($(VERSION))..."
$(MAKE) -C gateway-controller build VERSION=$(VERSION)

.PHONY: build-local-controller
build-local-controller: ## Build gateway-controller Docker image locally (faster)
@echo "Building controller Docker image locally ($(VERSION))..."
$(MAKE) -C gateway-controller build-local VERSION=$(VERSION)

.PHONY: build-controller-coverage
build-controller-coverage: ## Build coverage-instrumented gateway-controller Docker image
@echo "Building coverage-instrumented controller Docker image ($(VERSION))..."
$(MAKE) -C gateway-controller build-coverage-image VERSION=$(VERSION)

.PHONY: build-controller-debug
build-controller-debug: ## Build debug gateway-controller image for remote debugging
@echo "Building debug controller Docker image ($(VERSION))..."
$(MAKE) -C gateway-controller build-debug VERSION=$(VERSION)

.PHONY: build-gateway-builder
build-gateway-builder: ## Build gateway-builder Docker image
@echo "Building gateway-builder Docker image ($(VERSION))..."
$(MAKE) -C gateway-builder build VERSION=$(VERSION)

.PHONY: build-local-gateway-builder
build-local-gateway-builder: ## Build gateway-builder Docker image locally (faster)
@echo "Building gateway-builder Docker image locally ($(VERSION))..."
$(MAKE) -C gateway-builder build-local VERSION=$(VERSION)

.PHONY: build-gateway-runtime
build-gateway-runtime: ## Build gateway-runtime Docker image (Router + Policy Engine)
@echo "Building gateway-runtime Docker image ($(VERSION))..."
$(MAKE) -C gateway-runtime build VERSION=$(VERSION)

.PHONY: build-local-gateway-runtime
build-local-gateway-runtime: ## Build gateway-runtime Docker image locally (faster)
@echo "Building gateway-runtime Docker image locally ($(VERSION))..."
$(MAKE) -C gateway-runtime build-local VERSION=$(VERSION)

.PHONY: build-gateway-runtime-coverage
build-gateway-runtime-coverage: ## Build coverage-instrumented gateway-runtime Docker image
@echo "Building coverage-instrumented gateway-runtime Docker image ($(VERSION))..."
$(MAKE) -C gateway-runtime build-coverage-image VERSION=$(VERSION)

.PHONY: build-gateway-runtime-debug
build-gateway-runtime-debug: ## Build debug gateway-runtime image for remote debugging
@echo "Building debug gateway-runtime Docker image ($(VERSION))..."
$(MAKE) -C gateway-runtime build-debug VERSION=$(VERSION)

# Multi-arch Build and Push Targets
.PHONY: build-and-push-multiarch
build-and-push-multiarch: build-and-push-multiarch-controller build-and-push-multiarch-gateway-builder build-and-push-multiarch-gateway-runtime ## Build and push all gateway components for multiple architectures
Expand Down
98 changes: 98 additions & 0 deletions gateway/docker-compose.debug.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# --------------------------------------------------------------------
# Copyright (c) 2026, WSO2 LLC. (https://www.wso2.com).
#
# WSO2 LLC. licenses this file to you under the Apache License,
# Version 2.0 (the "License"); you may not use this file except
# in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# --------------------------------------------------------------------

# Compose overlay for remote debugging gateway components with dlv.
# Usage: docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
Comment on lines +19 to +20
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix the usage comment — this file is standalone, not an overlay.

As noted in the debug guide review, this file defines complete service definitions. Running it as an overlay on docker-compose.yaml would cause duplicate port bindings and conflicting configs.

Proposed fix 
 # Compose overlay for remote debugging gateway components with dlv.
-# Usage: docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
+# Usage: docker compose -f docker-compose.debug.yaml up
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Compose overlay for remote debugging gateway components with dlv.
# Usage: docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
# Compose overlay for remote debugging gateway components with dlv.
# Usage: docker compose -f docker-compose.debug.yaml up
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@gateway/docker-compose.debug.yaml` around lines 19 - 20, The header comment
("Compose overlay for remote debugging gateway components with dlv.") and the
usage line are incorrect — this YAML is a standalone docker-compose file, not an
overlay; update the top comment and the usage example to state it is a
standalone compose configuration for remote debugging and show the correct
command (e.g., "docker compose -f docker-compose.debug.yaml up") so users don't
mistakenly combine it with docker-compose.yaml and cause duplicate bindings;
keep the description mentioning dlv and debugging gateway components but remove
or replace "overlay" language and the "-f docker-compose.yaml -f" usage pattern.

#
# Prerequisites:
# 1. Build debug images:
# cd gateway-controller && make build-debug
# cd gateway-runtime && make build-debug
# 2. In VS Code Run & Debug:
# - "Gateway Controller (Remote)" (port 2345)
# - "Policy Engine (Remote)" (port 2346)

# NOTE: gateway-controller and gateway-runtime here duplicate docker-compose.yaml — keep in sync.

services:
gateway-controller:
image: ghcr.io/wso2/api-platform/gateway-controller-debug:latest
command: ["-config", "/etc/gateway-controller/config.toml"]
ports:
- "9090:9090" # REST API
- "9094:9092" # Admin API
- "18000:18000" # xDS gRPC (Router)
- "18001:18001" # xDS gRPC (Policy Engine)
- "9011:9091" # Metrics
- "2345:2345" # dlv remote debug port
environment:
- APIP_GW_CONTROLLER_STORAGE_TYPE=sqlite
- APIP_GW_CONTROLLER_STORAGE_SQLITE_PATH=./data/gateway.db
- APIP_GW_CONTROLLER_LOGGING_LEVEL=info
- APIP_GW_CONTROLLER_METRICS_PORT=9091
- APIP_GW_GATEWAY_REGISTRATION_TOKEN=${GATEWAY_REGISTRATION_TOKEN:-}
- APIP_GW_CONTROLPLANE_HOST=${GATEWAY_CONTROLPLANE_HOST:-host.docker.internal:9243}
volumes:
- controller-data:/app/data
- ./configs/config.toml:/etc/gateway-controller/config.toml:ro
- ./gateway-controller/certificates:/app/certificates
- ./gateway-controller/listener-certs:/app/listener-certs:ro
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- gateway-network
cap_add:
- SYS_PTRACE # required by dlv for process introspection
security_opt:
- seccomp:unconfined # dlv uses syscalls blocked by the default seccomp profile

gateway-runtime:
image: ghcr.io/wso2/api-platform/gateway-runtime-debug:latest
command: ["--pol.config", "/etc/policy-engine/config.toml"]
ports:
# Router (Envoy)
- "8080:8080" # HTTP ingress
- "8443:8443" # HTTPS ingress
- "8081:8081" # xDS-managed API listener
- "8082:8082" # WebSub Hub dynamic forward proxy
- "8083:8083" # WebSub Hub internal listener
- "9901:9901" # Envoy admin
# Policy Engine
- "9002:9002" # Admin API
- "9003:9003" # Metrics
- "2346:2346" # dlv remote debug port
environment:
- GATEWAY_CONTROLLER_HOST=gateway-controller
- MOESIF_KEY=${MOESIF_KEY:-}
- LOG_LEVEL=info
volumes:
- ./configs/config.toml:/etc/policy-engine/config.toml:ro
networks:
- gateway-network
cap_add:
- SYS_PTRACE # required by dlv for process introspection
security_opt:
- seccomp:unconfined # dlv uses syscalls blocked by the default seccomp profile

volumes:
controller-data:
driver: local

networks:
gateway-network:
driver: bridge
18 changes: 11 additions & 7 deletions gateway/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
# under the License.
# --------------------------------------------------------------------

# NOTE: gateway-controller and gateway-runtime are duplicated in docker-compose.debug.yaml — keep in sync.

services:
gateway-controller:
image: ghcr.io/wso2/api-platform/gateway-controller:0.8.8-SNAPSHOT
Expand Down Expand Up @@ -53,14 +55,16 @@ services:
cpus: 0.175
command: ["--pol.config", "/etc/policy-engine/config.toml"]
ports:
- "8080:8080" # HTTP traffic (Envoy)
- "8443:8443" # HTTPS traffic (Envoy)
# Router (Envoy)
- "8080:8080" # HTTP ingress
- "8443:8443" # HTTPS ingress
- "8081:8081" # xDS-managed API listener
- "8082:8082" # WebSub Hub dynamic forward proxy
- "8083:8083" # WebSub Hub internal listener
- "9901:9901" # Envoy admin
- "8081:8081"
- "8082:8082"
- "8083:8083"
- "9002:9002" # Policy Engine admin
- "9003:9003" # Policy Engine metrics
# Policy Engine
- "9002:9002" # Admin API
- "9003:9003" # Metrics
environment:
- GATEWAY_CONTROLLER_HOST=gateway-controller
- MOESIF_KEY=${MOESIF_KEY:-}
Expand Down
Loading
Loading