Skip to content

Optimize gateway tests #1102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Tharsanan1 wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Optimize gateway tests #1102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
754d59e
Improve config dump error response messaging
Tharsanan1 Feb 12, 2026
e0e5215
Optimize gateway integration tests by parallelizing mock builds and t…
Tharsanan1 Feb 12, 2026
8976893
Optimize workflow by building images once and sharing via artifacts
Tharsanan1 Feb 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 70 additions & 7 deletions .github/workflows/gateway-integration-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
- '.github/workflows/gateway-integration-test.yml'

jobs:
integration-test:
build-images:
runs-on: ubuntu-24.04
steps:
- name: Checkout code
Expand All @@ -20,6 +20,7 @@ jobs:
uses: actions/setup-go@v5
with:
go-version: '1.25'
cache: true

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
Expand All @@ -31,29 +32,91 @@ jobs:

- name: Build mock server images
run: |
for mock in mock-jwks mock-azure-content-safety mock-aws-bedrock-guardrail mock-embedding-provider mock-analytics-collector; do
mocks=("mock-jwks" "mock-azure-content-safety" "mock-aws-bedrock-guardrail" "mock-embedding-provider" "mock-analytics-collector")
for mock in "${mocks[@]}"; do
echo "Building $mock..."
docker build -t ghcr.io/wso2/api-platform/$mock:latest tests/mock-servers/$mock
docker build -t ghcr.io/wso2/api-platform/$mock:latest tests/mock-servers/$mock &
done
wait
echo "All mock images built."
Comment on lines +35 to +41
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Backgrounded build failures may be silently ignored.

Bash wait with no arguments returns the exit status of the last process waited for. If an earlier docker build fails but a later one succeeds, the step will pass and the missing image will cause a confusing failure later during tests.

Capture PIDs and check each exit code:

🐛 Proposed fix to propagate build failures 
          mocks=("mock-jwks" "mock-azure-content-safety" "mock-aws-bedrock-guardrail" "mock-embedding-provider" "mock-analytics-collector")
+         pids=()
          for mock in "${mocks[@]}"; do
            echo "Building $mock..."
            docker build -t ghcr.io/wso2/api-platform/$mock:latest tests/mock-servers/$mock &
+           pids+=($!)
          done
-         wait
+         failed=0
+         for pid in "${pids[@]}"; do
+           wait "$pid" || failed=1
+         done
+         if [ "$failed" -ne 0 ]; then
+           echo "One or more mock image builds failed."
+           exit 1
+         fi
          echo "All mock images built."
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
mocks=("mock-jwks" "mock-azure-content-safety" "mock-aws-bedrock-guardrail" "mock-embedding-provider" "mock-analytics-collector")
for mock in "${mocks[@]}"; do
echo "Building $mock..."
docker build -t ghcr.io/wso2/api-platform/$mock:latest tests/mock-servers/$mock
docker build -t ghcr.io/wso2/api-platform/$mock:latest tests/mock-servers/$mock &
done
wait
echo "All mock images built."
mocks=("mock-jwks" "mock-azure-content-safety" "mock-aws-bedrock-guardrail" "mock-embedding-provider" "mock-analytics-collector")
pids=()
for mock in "${mocks[@]}"; do
echo "Building $mock..."
docker build -t ghcr.io/wso2/api-platform/$mock:latest tests/mock-servers/$mock &
pids+=($!)
done
failed=0
for pid in "${pids[@]}"; do
wait "$pid" || failed=1
done
if [ "$failed" -ne 0 ]; then
echo "One or more mock image builds failed."
exit 1
fi
echo "All mock images built."
🤖 Prompt for AI Agents 
In @.github/workflows/gateway-integration-test.yml around lines 41 - 47, The
backgrounded docker builds in the for-loop using the mocks array may hide
failures because a plain wait only returns the last job's status; update the
loop that starts docker build (symbols: mocks array, the for mock in
"${mocks[@]}" loop, docker build) to record each background PID (e.g., push $!
into a pids array), then iterate over the pids and wait on each PID
individually, checking each exit code and failing the step (exit non-zero) if
any wait returns non-zero so build failures are propagated and the workflow
fails fast.


- name: Save images to artifact
run: |
docker save \
ghcr.io/wso2/api-platform/gateway-controller-coverage:test \
ghcr.io/wso2/api-platform/gateway-runtime-coverage:test \
ghcr.io/wso2/api-platform/mock-jwks:latest \
ghcr.io/wso2/api-platform/mock-azure-content-safety:latest \
ghcr.io/wso2/api-platform/mock-aws-bedrock-guardrail:latest \
ghcr.io/wso2/api-platform/mock-embedding-provider:latest \
ghcr.io/wso2/api-platform/mock-analytics-collector:latest \
| gzip > images.tar.gz

- name: Upload images
uses: actions/upload-artifact@v4
with:
name: docker-images
path: images.tar.gz
retention-days: 1

integration-test:
needs: build-images
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
group: [1, 2, 3, 4]
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.25'
cache: true

- name: Download images
uses: actions/download-artifact@v4
with:
name: docker-images

- name: Load images
run: |
gunzip -c images.tar.gz | docker load

- name: Run integration tests
- name: Run integration tests (Group ${{ matrix.group }})
run: |
case ${{ matrix.group }} in
1)
FEATURES="features/health.feature,features/metrics.feature,features/api_deploy.feature,features/mcp_deploy.feature,features/ratelimit.feature,features/jwt-auth.feature,features/cors.feature,features/word-count-guardrail.feature,features/sentence-count-guardrail.feature,features/url-guardrail.feature,features/regex-guardrail.feature"
;;
2)
FEATURES="features/prompt-decorator.feature,features/prompt-template.feature,features/pii-masking-regex.feature,features/json-schema-guardrail.feature,features/llm-provider-templates.feature,features/analytics-header-filter.feature,features/lazy-resources-xds.feature,features/content-length-guardrail.feature,features/azure-content-safety.feature,features/aws-bedrock-guardrail.feature,features/semantic-cache.feature"
;;
3)
FEATURES="features/semantic-prompt-guard.feature,features/modify-headers.feature,features/request-rewrite.feature,features/respond.feature,features/llm-provider.feature,features/certificates.feature,features/config-dump.feature,features/api-management.feature,features/api-error-responses.feature,features/list-policies.feature,features/api-keys.feature"
;;
4)
FEATURES="features/api-with-policies.feature,features/llm-proxies.feature,features/search-deployments.feature,features/policy-engine-admin.feature,features/cel-conditions.feature,features/analytics-basic.feature,features/backend_timeout.feature,features/llm-cost-based-ratelimit.feature,features/model-round-robin.feature,features/model-weighted-round-robin.feature"
;;
esac
cd gateway
make test-integration
IT_FEATURE_PATHS="$FEATURES" make test-integration

- name: Upload coverage report
uses: actions/upload-artifact@v4
if: always()
with:
name: coverage-report
name: coverage-report-group-${{ matrix.group }}
path: gateway/it/coverage/output
retention-days: 7

- name: Upload test reports
uses: actions/upload-artifact@v4
if: always()
with:
name: test-reports
name: test-reports-group-${{ matrix.group }}
path: gateway/it/reports/
retention-days: 7

Expand Down
2 changes: 1 addition & 1 deletion gateway/gateway-controller/pkg/api/handlers/handlers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2159,7 +2159,7 @@ func (s *APIServer) GetConfigDump(c *gin.Context) {
log.Error("Failed to retrieve configuration dump", slog.Any("error", err))
c.JSON(http.StatusInternalServerError, api.ErrorResponse{
Status: "error",
Message: "Failed to retrieve certificates",
Message: err.Error(),
})
return
}
Expand Down
Loading