feat: enhance wait_for_ready() on policies. by crstrn13 · Pull Request #847 · Kuadrant/testsuite

crstrn13 · 2026-01-19T15:39:15Z

Description

Introduces a topology registry system for tracking Gateway API resource relationships (Gateways, HTTPRoutes, Policies)
Adds comprehensive metrics infrastructure with factory pattern for different gateway exposers (LoadBalancer, OpenShift)
Refactors policy section handling with generic Section base class to eliminate code duplication and prevent empty dict creation
Enhances wait_for_ready() with Envoy-aware waiting mechanism using metrics validation

Changes

New Infrastructure

Topology Registry (testsuite/core/topology.py): 493-line system for tracking Gateway API resource relationships and dependencies
- TopologyRegistry singleton for global resource tracking
- TopologyNode for representing resources and their relationships (targets, targeted_by, children, parent)
- @topology decorator for automatic resource registration
Metrics Factory (testsuite/core/metrics_factory.py): Factory pattern for creating GatewayMetrics implementations based on exposer type
Gateway Metrics (testsuite/gateway/metrics.py): Abstract base class and implementations for LoadBalancer and OpenShift gateway metrics
Policy Metric Validator (testsuite/kuadrant/policy/metric_validator.py): Base class for validating policy-related metrics

Refactoring

Section Pattern (testsuite/kuadrant/policy/__init__.py): Generic Section base class with:
- Smart get_section() navigation for defaults/overrides and auth sections
- Generic add_to_spec() helper for dataclass conversion
- committed property delegation and modify_and_apply() support
- @modify decorator on strategy() method
RateLimitPolicy (testsuite/kuadrant/policy/rate_limit.py): Uses RateLimitSection subclass, eliminates code duplication in add_limit()
AuthPolicy (testsuite/kuadrant/policy/authorization/auth_policy.py): Uses AuthSection combining Section + AuthConfig
Auth Sections (testsuite/kuadrant/policy/authorization/sections.py): Updated to inherit from base Section class

Enhancements

Gateway API Gateway (testsuite/gateway/gateway_api/gateway.py): Added wait_for_ready() with Envoy readiness validation via metrics
Gateway API Route (testsuite/gateway/gateway_api/route.py): Added wait_for_ready() delegating to gateway
Service (testsuite/kubernetes/service.py): Added service_url() method
Gateway Exposers: Added get_metrics() method to Exposer protocol
Envoy Gateway (testsuite/gateway/envoy/__init__.py): Implements metrics support

Tests

Updated test fixtures to use new topology registration
Added wait_for_ready() calls in multiple test conftest files to ensure Envoy readiness before running tests
Updated 28 test files to work with new infrastructure

Verification steps

All modified test files should be verified to ensure they work with the new infrastructure:

# Run all affected tests
poetry run pytest -vv -n4 \
  testsuite/tests/conftest.py \
  testsuite/tests/singlecluster/conftest.py \
  testsuite/tests/singlecluster/defaults/merge/auth_policy/conftest.py \
  testsuite/tests/singlecluster/defaults/merge/rate_limit/conftest.py \
  testsuite/tests/singlecluster/defaults/merge/rate_limit/same_target/test_ab_strategy.py \
  testsuite/tests/singlecluster/defaults/test_section_targeting.py \
  testsuite/tests/singlecluster/extensions/oidc_policy/conftest.py \
  testsuite/tests/singlecluster/extensions/plan_policy/test_plan_policy.py \
  testsuite/tests/singlecluster/extensions/telemetry_policy/conftest.py \
  testsuite/tests/singlecluster/gateway/authpolicy/test_authpolicy_section_targeting_gateway.py \
  testsuite/tests/singlecluster/gateway/authpolicy/test_authpolicy_section_targeting_http_route.py \
  testsuite/tests/singlecluster/gateway/conftest.py \
  testsuite/tests/singlecluster/gateway/dnspolicy/health_check/test_additional_headers.py

# Or run full test suite
make test

Closes: #846

testsuite/tests/singlecluster/conftest.py

crstrn13 · 2026-01-23T09:24:36Z

/make smoke

github-actions · 2026-01-23T09:25:09Z

Test run completed (make smoke) and can be found here

Short Test Summary

=========================== short test summary info ============================
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] - AssertionError: AuthPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
================== 16 warnings, 5 errors in 396.01s (0:06:36) ==================

Full Output

poetry lock
Creating virtualenv kuadrant-testsuite-6EPd9kcO-py3.12 in /home/runner/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

Writing lock file
Installing dependencies from lock file

Package operations: 51 installs, 0 updates, 0 removals

  - Installing aiofiles (25.1.0)
  - Installing anyio (4.12.1)
  - Installing apyproxy (0.2.2)
  - Installing backoff (2.2.1)
  - Installing build (1.4.0)
  - Installing certifi (2026.1.4)
  - Installing cffi (2.0.0)
  - Installing charset-normalizer (3.4.4)
  - Installing cryptography (46.0.3)
  - Installing deprecation (2.1.0)
  - Installing dnspython (2.8.0)
  - Installing dynaconf (3.2.12)
  - Installing execnet (2.1.2)
  - Installing greenlet (3.3.0)
  - Installing h11 (0.16.0)
  - Installing h2 (4.3.0)
  - Installing hpack (4.1.0)
  - Installing httpcore (1.0.9)
  - Installing httpx (0.28.1)
  - Installing hyperframe (6.1.0)
  - Installing idna (3.11)
  - Installing iniconfig (2.3.0)
  - Installing jinja2 (3.1.6)
  - Installing jwcrypto (1.5.6)
  - Installing lxml (6.0.2)
  - Installing markupsafe (3.0.3)
  - Installing openshift-client (2.0.5)
  - Installing packaging (26.0)
  - Installing playwright (1.57.0)
  - Installing pluggy (1.6.0)
  - Installing pycparser (3.0)
  - Installing pyee (13.0.0)
  - Installing pygments (2.19.2)
  - Installing pyjwt (2.10.1)
  - Installing pyproject-hooks (1.2.0)
  - Installing pytest (9.0.2)
  - Installing pytest-base-url (2.1.0)
  - Installing pytest-html (4.2.0)
  - Installing pytest-metadata (3.1.1)
  - Installing pytest-playwright (0.7.2)
  - Installing pytest-xdist (3.8.0)
  - Installing python-keycloak (7.0.2)
  - Installing python-slugify (8.0.4)
  - Installing pyyaml (6.0.3)
  - Installing requests (2.32.5)
  - Installing requests-toolbelt (1.0.0)
  - Installing six (1.17.0)
  - Installing text-unidecode (1.3)
  - Installing typing-extensions (4.15.0)
  - Installing urllib3 (2.6.3)
  - Installing weakget (1.0)
poetry run python -m pytest --tb=short -o cache_dir=./.pytest_cache.smoke -n4 -m 'smoke' --dist loadfile --enforce -vv testsuite/tests/
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
cachedir: .pytest_cache.smoke
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.11.0-1018-azure-x86_64-with-glibc2.39', 'Packages': {'pytest': '9.0.2', 'pluggy': '1.6.0'}, 'Plugins': {'base-url': '2.1.0', 'playwright': '0.7.2', 'xdist': '3.8.0', 'html': '4.2.0', 'anyio': '4.12.1', 'metadata': '3.1.1'}, 'CI': 'true', 'JAVA_HOME': '/usr/lib/jvm/temurin-17-jdk-amd64', 'Base URL': '', 'Kuadrant': []}

rootdir: /home/runner/work/testsuite/testsuite
configfile: pyproject.toml
plugins: base-url-2.1.0, playwright-0.7.2, xdist-3.8.0, html-4.2.0, anyio-4.12.1, metadata-3.1.1
created: 4/4 workers
4 workers [5 items]

scheduling tests via LoadFileScheduling

testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness 
testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] 
testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] 
[gw0] [ 20%] ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness 
testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls 
[gw0] [ 40%] ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls 
[gw2] [ 60%] ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] 
testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] 
[gw1] [ 80%] ERROR testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] 
[gw2] [100%] ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] 

==================================== ERRORS ====================================
___________________ ERROR at setup of test_gateway_readiness ___________________
[gw0] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
_________________ ERROR at setup of test_gateway_basic_dns_tls _________________
[gw0] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
_________ ERROR at setup of test_limit[2 requests every 15 sec-route] __________
[gw2] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
testsuite/tests/singlecluster/limitador/conftest.py:18: in commit
    rate_limit.wait_for_ready()
testsuite/kuadrant/policy/rate_limit.py:98: in wait_for_ready
    super().wait_for_ready()
testsuite/kuadrant/policy/__init__.py:131: in wait_for_ready
    self.wait_for_envoy_applied()
testsuite/kuadrant/policy/__init__.py:52: in wait_for_envoy_applied
    assert success, (
           ^^^^^^^
E   AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
------------------------------ Captured log setup ------------------------------
09:29:45 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:29:46 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:29:47 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:29:49 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:29:52 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:29:57 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:30:05 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:30:18 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:30:20 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:30:21 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:30:22 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:30:24 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:30:27 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:30:32 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:30:40 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:30:53 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:30:58 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:30:59 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:31:00 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:31:02 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:31:05 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:31:10 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:31:18 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:31:31 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:31:36 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:31:37 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:31:38 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:31:40 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:31:43 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:31:48 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:31:56 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:32:09 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:32:14 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:32:15 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:32:16 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:32:18 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:32:21 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:32:26 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:32:34 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:32:47 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
________________ ERROR at setup of test_auth_identity[keycloak] ________________
[gw1] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
testsuite/tests/singlecluster/authorino/conftest.py:57: in commit
    authorization.wait_for_ready()
testsuite/kuadrant/policy/__init__.py:131: in wait_for_ready
    self.wait_for_envoy_applied()
testsuite/kuadrant/policy/__init__.py:52: in wait_for_envoy_applied
    assert success, (
           ^^^^^^^
E   AssertionError: AuthPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
------------------------------ Captured log setup ------------------------------
09:29:47 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:29:48 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:29:49 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:29:51 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:29:54 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:29:59 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:30:07 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:30:20 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:30:22 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:30:23 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:30:24 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:30:26 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:30:29 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:30:34 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:30:42 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:30:55 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:31:00 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:31:01 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:31:02 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:31:04 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:31:07 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:31:12 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:31:20 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:31:33 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:31:38 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:31:39 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:31:40 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:31:42 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:31:45 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:31:50 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:31:58 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:32:11 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:32:16 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:32:17 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:32:18 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:32:20 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:32:23 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:32:28 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:32:36 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:32:49 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
________ ERROR at setup of test_limit[2 requests every 15 sec-gateway] _________
[gw2] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
testsuite/tests/singlecluster/limitador/conftest.py:18: in commit
    rate_limit.wait_for_ready()
testsuite/kuadrant/policy/rate_limit.py:98: in wait_for_ready
    super().wait_for_ready()
testsuite/kuadrant/policy/__init__.py:131: in wait_for_ready
    self.wait_for_envoy_applied()
testsuite/kuadrant/policy/__init__.py:52: in wait_for_envoy_applied
    assert success, (
           ^^^^^^^
E   AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
------------------------------ Captured log setup ------------------------------
09:32:52 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:32:53 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:32:54 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:32:56 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:32:59 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:33:04 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:33:12 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:33:25 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:33:26 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:33:27 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:33:28 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:33:30 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:33:33 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:33:38 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:33:46 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:33:59 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:34:04 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:34:05 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:34:06 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:34:08 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:34:11 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:34:16 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:34:24 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:34:37 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:34:42 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:34:43 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:34:44 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:34:46 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:34:49 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:34:54 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:35:02 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:35:15 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
09:35:20 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:35:21 +0000 INFO:backoff:Backing off request(...) for 1.0s (Result[error=[Errno -2] Name or service not known])
09:35:22 +0000 INFO:backoff:Backing off request(...) for 2.0s (Result[error=[Errno -2] Name or service not known])
09:35:24 +0000 INFO:backoff:Backing off request(...) for 3.0s (Result[error=[Errno -2] Name or service not known])
09:35:27 +0000 INFO:backoff:Backing off request(...) for 5.0s (Result[error=[Errno -2] Name or service not known])
09:35:32 +0000 INFO:backoff:Backing off request(...) for 8.0s (Result[error=[Errno -2] Name or service not known])
09:35:40 +0000 INFO:backoff:Backing off request(...) for 13.0s (Result[error=[Errno -2] Name or service not known])
09:35:53 +0000 ERROR:backoff:Giving up request(...) after 8 tries (Result[error=[Errno -2] Name or service not known])
=============================== warnings summary ===============================
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] - AssertionError: AuthPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
================== 16 warnings, 5 errors in 396.01s (0:06:36) ==================
make: *** [Makefile:33: smoke] Error 1

azgabur · 2026-01-27T11:29:04Z

/make smoke

github-actions · 2026-01-27T11:29:15Z

Test run completed (make smoke) and can be found here

Short Test Summary

=========================== short test summary info ============================
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] - AssertionError: AuthPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
================== 16 warnings, 5 errors in 270.01s (0:04:30) ==================

Full Output

poetry lock
Creating virtualenv kuadrant-testsuite-6EPd9kcO-py3.12 in /home/runner/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

Writing lock file
Installing dependencies from lock file

Package operations: 51 installs, 0 updates, 0 removals

  - Installing aiofiles (25.1.0)
  - Installing anyio (4.12.1)
  - Installing apyproxy (0.2.2)
  - Installing backoff (2.2.1)
  - Installing build (1.4.0)
  - Installing certifi (2026.1.4)
  - Installing cffi (2.0.0)
  - Installing charset-normalizer (3.4.4)
  - Installing cryptography (46.0.3)
  - Installing deprecation (2.1.0)
  - Installing dnspython (2.8.0)
  - Installing dynaconf (3.2.12)
  - Installing execnet (2.1.2)
  - Installing greenlet (3.3.1)
  - Installing h11 (0.16.0)
  - Installing h2 (4.3.0)
  - Installing hpack (4.1.0)
  - Installing httpcore (1.0.9)
  - Installing httpx (0.28.1)
  - Installing hyperframe (6.1.0)
  - Installing idna (3.11)
  - Installing iniconfig (2.3.0)
  - Installing jinja2 (3.1.6)
  - Installing jwcrypto (1.5.6)
  - Installing lxml (6.0.2)
  - Installing markupsafe (3.0.3)
  - Installing openshift-client (2.0.5)
  - Installing packaging (26.0)
  - Installing playwright (1.57.0)
  - Installing pluggy (1.6.0)
  - Installing pycparser (3.0)
  - Installing pyee (13.0.0)
  - Installing pygments (2.19.2)
  - Installing pyjwt (2.10.1)
  - Installing pyproject-hooks (1.2.0)
  - Installing pytest (9.0.2)
  - Installing pytest-base-url (2.1.0)
  - Installing pytest-html (4.2.0)
  - Installing pytest-metadata (3.1.1)
  - Installing pytest-playwright (0.7.2)
  - Installing pytest-xdist (3.8.0)
  - Installing python-keycloak (7.0.2)
  - Installing python-slugify (8.0.4)
  - Installing pyyaml (6.0.3)
  - Installing requests (2.32.5)
  - Installing requests-toolbelt (1.0.0)
  - Installing six (1.17.0)
  - Installing text-unidecode (1.3)
  - Installing typing-extensions (4.15.0)
  - Installing urllib3 (2.6.3)
  - Installing weakget (1.0)
poetry run python -m pytest --tb=short -o cache_dir=./.pytest_cache.smoke -n4 -m 'smoke' --dist loadfile --enforce -vv testsuite/tests/
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
cachedir: .pytest_cache.smoke
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.11.0-1018-azure-x86_64-with-glibc2.39', 'Packages': {'pytest': '9.0.2', 'pluggy': '1.6.0'}, 'Plugins': {'base-url': '2.1.0', 'playwright': '0.7.2', 'xdist': '3.8.0', 'html': '4.2.0', 'anyio': '4.12.1', 'metadata': '3.1.1'}, 'CI': 'true', 'JAVA_HOME': '/usr/lib/jvm/temurin-17-jdk-amd64', 'Base URL': '', 'Kuadrant': []}

rootdir: /home/runner/work/testsuite/testsuite
configfile: pyproject.toml
plugins: base-url-2.1.0, playwright-0.7.2, xdist-3.8.0, html-4.2.0, anyio-4.12.1, metadata-3.1.1
created: 4/4 workers
4 workers [5 items]

scheduling tests via LoadFileScheduling

testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] 
testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] 
testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness 
[gw2] [ 20%] ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness 
testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls 
[gw2] [ 40%] ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls 
[gw3] [ 60%] ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] 
testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] 
[gw0] [ 80%] ERROR testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] 
[gw3] [100%] ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] 

==================================== ERRORS ====================================
___________________ ERROR at setup of test_gateway_readiness ___________________
[gw2] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
_________________ ERROR at setup of test_gateway_basic_dns_tls _________________
[gw2] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
_________ ERROR at setup of test_limit[2 requests every 15 sec-route] __________
[gw3] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
testsuite/tests/singlecluster/limitador/conftest.py:18: in commit
    rate_limit.wait_for_ready()
testsuite/kuadrant/policy/rate_limit.py:98: in wait_for_ready
    super().wait_for_ready()
testsuite/kuadrant/policy/__init__.py:131: in wait_for_ready
    self.wait_for_envoy_applied()
testsuite/kuadrant/policy/__init__.py:52: in wait_for_envoy_applied
    assert success, (
           ^^^^^^^
E   AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
------------------------------ Captured log setup ------------------------------
11:34:24 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:26 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:31 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:36 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:41 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:46 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:51 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:56 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:01 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:06 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:11 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:17 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:22 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:27 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:32 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:37 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:42 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:47 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:52 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:57 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:02 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:08 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:13 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:18 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:23 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
________________ ERROR at setup of test_auth_identity[keycloak] ________________
[gw0] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
testsuite/tests/singlecluster/authorino/conftest.py:57: in commit
    authorization.wait_for_ready()
testsuite/kuadrant/policy/__init__.py:131: in wait_for_ready
    self.wait_for_envoy_applied()
testsuite/kuadrant/policy/__init__.py:52: in wait_for_envoy_applied
    assert success, (
           ^^^^^^^
E   AssertionError: AuthPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
------------------------------ Captured log setup ------------------------------
11:34:27 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:30 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:35 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:40 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:45 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:50 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:34:55 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:00 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:05 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:10 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:15 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:21 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:26 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:31 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:36 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:41 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:46 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:51 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:35:56 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:01 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:06 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:11 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:17 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:22 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:27 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.210:15020/stats/prometheus "HTTP/1.1 200 OK"
________ ERROR at setup of test_limit[2 requests every 15 sec-gateway] _________
[gw3] linux -- Python 3.12.3 /home/runner/.cache/pypoetry/virtualenvs/kuadrant-testsuite-6EPd9kcO-py3.12/bin/python
testsuite/tests/singlecluster/limitador/conftest.py:18: in commit
    rate_limit.wait_for_ready()
testsuite/kuadrant/policy/rate_limit.py:98: in wait_for_ready
    super().wait_for_ready()
testsuite/kuadrant/policy/__init__.py:131: in wait_for_ready
    self.wait_for_envoy_applied()
testsuite/kuadrant/policy/__init__.py:52: in wait_for_envoy_applied
    assert success, (
           ^^^^^^^
E   AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
------------------------------ Captured log setup ------------------------------
11:36:28 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:29 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:34 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:40 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:45 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:50 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:36:55 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:00 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:05 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:10 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:15 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:20 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:25 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:31 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:36 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:41 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:46 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:51 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:37:56 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:38:01 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:38:06 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:38:11 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:38:16 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:38:21 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
11:38:27 +0000 INFO:httpx:HTTP Request: GET http://172.18.255.209:15020/stats/prometheus "HTTP/1.1 200 OK"
=============================== warnings summary ===============================
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_dns_policy.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/dnstls/test_tls_policy.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/test_auth_policy.py:7: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7
  /home/runner/work/testsuite/testsuite/testsuite/tests/singlecluster/ui/console_plugin/policies/test_rate_limit_policy.py:7: PytestUnknownMarkWarning: Unknown pytest.mark.ui - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    pytestmark = [pytest.mark.ui]

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_readiness - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/gateway/test_basic.py::test_gateway_basic_dns_tls - Failed: ScopeMismatch: You tried to access the module scoped fixture gateway with a session scoped request object. Requesting fixture stack:
testsuite/tests/singlecluster/conftest.py:131:  def gateway_metrics_service(request, gateway, cluster, blame, kuadrant, exposer, label)
Requested fixture:
testsuite/tests/singlecluster/gateway/conftest.py:14:  def gateway(request, cluster, blame, wildcard_domain, module_label)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-route] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/authorino/identity/auth/test_auth_identity.py::test_auth_identity[keycloak] - AssertionError: AuthPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
ERROR testsuite/tests/singlecluster/limitador/test_basic_limit.py::test_limit[2 requests every 15 sec-gateway] - AssertionError: RateLimitPolicy was ready in K8s but did not get applied in Envoy within 120s (initial kuadrant_configs: 0)
================== 16 warnings, 5 errors in 270.01s (0:04:30) ==================
make: *** [Makefile:33: smoke] Error 1

azgabur

This PR has a lot of strange refactors I do not understand why are needed to achieve the PR's goal.

The suggested way to extend Policy class to add the wait for kuadrant_config update seems bit hack-y to me. The metrics are exposed in Gateway, so I recommend moving the code which handles the update check there. The Policies which are being updated/applied could have reference to their gateway stored as a field (similar how they store "cluster") and use it inside commit (to get current kuadrant_config generation number) and wait_for_* methods (to check the number was incremented).
I can see the problem in which we do commit fixture. In some situations (for example testsuite/tests/multicluster/coredns/two_clusters/conftest.py) there are multiple policies commited before their wait_for_ready methods are called. Meaning you cant expect that .commit() and .wait_for_ready() methods will be called right after each other so the generation field could be incremented by something else then the current policy waiting.

I do not know how the kuadrant_config structure looks like, but if it contains field about generation of specific policy being applied, you could use generation field of the policy similar how the current wait_for_ready works in Policy to wait until the gateway has the field in the metric.

In any case, this check seems like something that could be done on the Operator side and used to set the Ready/Enforced status more correctly, I would suggest that as a issue and have the implementation of such check in testsuite only if they reject implementing it.

azgabur · 2026-01-27T11:34:14Z

testsuite/gateway/envoy/__init__.py

Why rename name -> _name ? And create duplicate name property?

azgabur · 2026-01-27T11:41:19Z

testsuite/gateway/__init__.py

        """Returns TLS cert bound to this Gateway, if the Gateway does not use TLS, returns None"""

+    @abstractmethod
+    def name(self) -> str:


There is already service_name

azgabur · 2026-01-27T12:15:46Z

testsuite/kuadrant/extensions/plan_policy.py

 class PlanPolicy(Policy):
    """PlanPolicy object, used for applying plan-based policies to a Gateway/HTTPRoute"""

-    def __init__(self, *args, **kwargs):


Unrelated refactor as well as in testsuite/kuadrant/extensions/telemetry_policy.py

azgabur · 2026-01-27T12:22:21Z

testsuite/tests/singlecluster/authorino/identity/plain/test_jwt_plain_identity.py

 def gateway(request, authorino, cluster, blame, module_label, testconfig, keycloak):
    """Deploys Envoy with additional JWT plain identity test setup"""
-    envoy = JwtEnvoy(
+    envoy = JwtEnvoy(  # pylint: disable=abstract-class-instantiated


JwtEnvoy is not abstract class, why is this disable here?

azgabur · 2026-01-27T14:01:58Z

testsuite/gateway/exposers.py

        self._hostname = hostname
-        self.verify = verify
        self.ip_getter = ip_getter
+        self.verify = verify


why change order?

it wasn't matching the parameters order.

azgabur · 2026-01-27T14:45:24Z

testsuite/tests/singlecluster/authorino/operator/clusterwide/conftest.py


 @pytest.fixture(scope="module", autouse=True)
-def commit(request, commit, authorization2):  # pylint: disable=unused-argument
+def commit(request, authorization2):  # pylint: disable=unused-argument


Why remove parent commit fixture here?

azgabur · 2026-01-27T14:46:06Z

testsuite/tests/singlecluster/authorino/operator/conftest.py

 def gateway(request, authorino, cluster, blame, label, testconfig) -> Envoy:
    """Deploys Envoy that wires up the Backend behind the reverse-proxy and Authorino instance"""
-    gw = Envoy(
+    gw = Envoy(  # pylint: disable=abstract-class-instantiated


Envoy is not abstract class, why is this disable here?

azgabur · 2026-01-27T14:46:53Z

testsuite/tests/singlecluster/authorino/wristband/conftest.py


 @pytest.fixture(scope="module", autouse=True)
-def commit(request, commit, wristband_authorization):  # pylint: disable=unused-argument
+def commit(request, wristband_authorization):  # pylint: disable=unused-argument


Why remove parent commit fixture here?

azgabur · 2026-01-27T15:11:01Z

testsuite/tests/singlecluster/conftest.py

+    metrics_service.commit()
+    metrics_service.wait_for_ready()
+
+    return exposer.expose_hostname(blame("metrics"), metrics_service)


You dont need to create whole class MetricsServiceGateway just to run exposer on it. In the worst case this will create loadbalanced service targeting loadbalanced service doubling infrastructure strain which now mostly causes the sporadic failures.

I suggest using the Gateway's loadbalanced service which already exists to expose the port you need (15020) for the metrics. See https://istio.io/latest/docs/tasks/observability/metrics/secure-metrics/#secure-metrics-for-gateways
This will avoid whole exposing hassle.

azgabur · 2026-01-27T15:12:32Z

testsuite/tests/singlecluster/conftest.py

+    hostname,
+    backend,
+    module_label,
+    gateway_metrics_service,  # pylint: disable=unused-argument


Why add it, if its not used

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

crstrn13 force-pushed the envoy_wait_mixin branch 2 times, most recently from 80700f6 to 24acf6d Compare January 20, 2026 08:31

crstrn13 self-assigned this Jan 20, 2026

crstrn13 force-pushed the envoy_wait_mixin branch from 24acf6d to ddc603e Compare January 20, 2026 08:38

crstrn13 requested review from averevki and azgabur January 20, 2026 09:10

crstrn13 force-pushed the envoy_wait_mixin branch from c3df06a to 4f11377 Compare January 20, 2026 09:16

azgabur reviewed Jan 20, 2026

View reviewed changes

testsuite/tests/singlecluster/conftest.py Outdated Show resolved Hide resolved

crstrn13 force-pushed the envoy_wait_mixin branch 3 times, most recently from 9304fb3 to 2325864 Compare January 22, 2026 16:24

crstrn13 requested review from azgabur and silvi-t January 23, 2026 15:58

azgabur reviewed Jan 27, 2026

View reviewed changes

azgabur mentioned this pull request Mar 10, 2026

Brief incorrect ready status on Policies Kuadrant/kuadrant-operator#1809

Open

crstrn13 force-pushed the envoy_wait_mixin branch 2 times, most recently from bd66428 to a82f6ad Compare March 13, 2026 16:27

crstrn13 requested a review from azgabur March 13, 2026 16:35

crstrn13 marked this pull request as draft March 14, 2026 09:57

crstrn13 force-pushed the envoy_wait_mixin branch 4 times, most recently from d692c8d to 0b52dc5 Compare March 14, 2026 19:59

easy fix

f14e9ad

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

crstrn13 force-pushed the envoy_wait_mixin branch from 0b52dc5 to 7125693 Compare March 14, 2026 22:17

fix: also added rlp.

36f7ece

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

crstrn13 added 18 commits March 15, 2026 00:01

refactor: policy spec.

494d7c3

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

refactor: split authconfig inheritance from authpolicy.

4db265c

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

refactor: comsetic changes.

08ff08b

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: authorino tests.

ef04cf5

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: bug fix.

7b4c6af

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: more fixes.

acdba0c

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

feat: make local-setup.

6dfa4a7

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: cosmetic changes.

bea610a

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: add cluster issuer.

29723be

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: add cluster issuer.

743d69c

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: added aws creds secret creation for dns.

76ef50e

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: cosmetic changes.

a935dfa

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

feat: added env patch for kuadrant-operator.

df9d917

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: env patch on kuadrant-operator.

b81f9d8

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: cosmetic changes.

20c2745

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: cosmetic changes.

a3a6ab1

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

revert: istio versions - 1.27 causes restart

36d2e9a

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

feat: enhanced wait_for_ready()

6b44322

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

crstrn13 force-pushed the envoy_wait_mixin branch from 7125693 to 6b44322 Compare March 14, 2026 23:28

crstrn13 added 2 commits March 15, 2026 11:05

fix: some more fixes.

d6fb2ed

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

fix: cosmetic changes.

15ea4a9

Signed-off-by: Alexander Cristurean <acristur@redhat.com>

Conversation

crstrn13 commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

New Infrastructure

Refactoring

Enhancements

Tests

Verification steps

Uh oh!

Uh oh!

crstrn13 commented Jan 23, 2026

Uh oh!

github-actions bot commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azgabur commented Jan 27, 2026

Uh oh!

github-actions bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azgabur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

crstrn13 commented Jan 19, 2026 •

edited

Loading

github-actions bot commented Jan 23, 2026 •

edited

Loading

github-actions bot commented Jan 27, 2026 •

edited

Loading