[Proposal] Supporting Endpoint Timeouts in the API Gateway

[O-sura]

Problem

The current API definition does not allow users to configure endpoint-level timeouts, which limits the gateway’s ability to enforce resiliency when upstream services are slow or unresponsive. Without explicit endpoint timeouts, requests may occupy gateway resources indefinitely, increasing the risk of cascading failures. As a result, we need a well-defined and intuitive configuration model that exposes endpoint timeouts while mapping cleanly to Envoy’s routing capabilities.

Proposed Solution

Introduce support for endpoint timeouts at both the API level and the operation level. At the API level, users can define an endpoint timeout that applies to all operations as a default. At the operation level, users can optionally override this value for specific endpoints that require different latency characteristics. For each operation, the gateway resolves a single effective timeout using the precedence operation-level → API-level → gateway default, and applies it to the corresponding Envoy route. A gateway-level default endpoint timeout is defined in the gateway configuration and acts as a safety net when no timeout is specified by the API author

API-level Endpoint Timeout (applies for all operations)

spec:
  requestTimeout: 2s

Operation-level Endpoint Timeout (override)

operations:
  - method: GET
    path: /{country_code}/{city}
    requestTimeout: 1s

  - method: GET
    path: /alerts/active
    requestTimeout: 5s

Benefits of proposed approach:

• Exposes endpoint timeout semantics clearly to API users
• Allows fine-grained control per endpoint while keeping sensible defaults
• Prevents indefinite waits and improves gateway resiliency
• Keeps the configuration extensible for future resiliency features

[BLasan]

Hi @O-sura ,

For upstream resiliency (backend), we need more granularity than simply configuring the requestTimeout. For example, we may need to configure circuit breakers, retries with backoff, etc. I would prefer introducing a separate object (or property) to configure all these resiliency-related parameters.

WDYT, @renuka-fernando?

Thanks!

[Krishanx92]

Let's proceed with upstream definition #369

[pubudu538]

We have other timeout values as well. Shall we make sure they are also captured here - https://apim.docs.wso2.com/en/4.2.0/deploy-and-publish/deploy-on-gateway/choreo-connect/endpoints/resiliency/timeout/

[renuka-fernando]

In envoy timeout is defined at the route level. In API YAML, if we configure this at the upstream level, this can lead to an issue when we are going to support the dynamic endpoint feature.

version: api-platform.wso2.com/v1alpha1
kind: HttpRestApi
metadata:
  name: weather-api
spec:
  displayName: Weather API
  version: v1.0 #
  context: /weather
  upstreamDefinitions:
    - name: my-upstream-1
      loadBalancer:
        strategy: ROUND_ROBIN
      timeout:
        connect: 5s
        request: 30s
        idle: 60s
      retry:
        attempts: 3
        conditions: [5xx, timeout, connection-failure]
        backoff: exponential
      upstreams: # Multiple upstreams
        - urls:
            - http://prod-backend-1:5000/api/v2
          weight: 80
        - urls:
            - http://sandbox-backend-1:5000/api/v2
          weight: 20

From a policy if the upstream my-upstream-1 is selected, there is no way to configure Envoy Route with the value defined here.

Envoy Route Action

{
  "cluster": ...,
  "cluster_header": ...,
  "weighted_clusters": {...},
  "cluster_specifier_plugin": ...,
  "inline_cluster_specifier_plugin": {...},
  "cluster_not_found_response_code": ...,
  "metadata_match": {...},
  "prefix_rewrite": ...,
  "regex_rewrite": {...},
  "path_rewrite_policy": {...},
  "path_rewrite": ...,
  "host_rewrite_literal": ...,
  "auto_host_rewrite": {...},
  "host_rewrite_header": ...,
  "host_rewrite_path_regex": {...},
  "host_rewrite": ...,
  "append_x_forwarded_host": ...,
  "timeout": {...}, ### TIMEOUT
  "idle_timeout": {...},
  "flush_timeout": {...},
  "early_data_policy": {...},
  "retry_policy": {...},
  "request_mirror_policies": [],
  "priority": ...,
  "rate_limits": [],
  "include_vh_rate_limits": {...},
  "hash_policy": [],
  "cors": {...},
  "max_grpc_timeout": {...},
  "grpc_timeout_offset": {...},
  "upgrade_configs": [],
  "internal_redirect_policy": {...},
  "internal_redirect_action": ...,
  "max_internal_redirects": {...},
  "hedge_policy": {...},
  "max_stream_duration": {...}
}

Suggestion

API Level and Route Level resilience field to define these values at the Envoy Route level. With this approach, we are keeping API Operation <-> Envoy Route mapping and API Upstream <-> Envoy Cluster.

resilience:
  timeout: 15s
  idle_timeout: 0s
  retry: {...}

[renuka-fernando]

cc: @nuwand, @pubudu538, @malinthaprasan, @Krishanx92

[renuka-fernando]

config.cluster.v3.Cluster

{
  "transport_socket_matches": [],
  "name": ...,
  "alt_stat_name": ...,
  "type": ...,
  "cluster_type": {...},
  "eds_cluster_config": {...},
  "connect_timeout": {...},
   ...
}

Connect timeout is a cluster-level config, so we can define this in API YAML, upstream config.

version: api-platform.wso2.com/v1alpha1
kind: HttpRestApi
metadata:
  name: weather-api
spec:
  displayName: Weather API
  version: v1.0
  context: /weather
  upstreamDefinitions:
    - name: my-upstream-1
      timeout:
        connect: 10s # Connect Timeout
      loadBalancer:
        strategy: ROUND_ROBIN
      upstreams:
        - urls:
            - http://prod-backend-1:5000/api/v2
          weight: 80
        - urls:
            - http://sandbox-backend-1:5000/api/v2
          weight: 20