[GATEWAY V2]: Bifurcate connect / connection-acquire timeout between Gateway V1 and Gateway V2 endpoints.

[jeet1995]

Problem

When thin client (Gateway V2) is enabled, both metadata requests (port 443, GW V1) and data-plane requests (port 10250, GW V2 HTTP/2) share the same CONNECT_TIMEOUT_MILLIS of 45s. If the thin client proxy on port 10250 is unreachable, the SDK waits 45s per connect attempt before failing — far too long for a data-plane path that should fail fast and trigger regional failover.

Solution

Bifurcate CONNECT_TIMEOUT_MILLIS at the Reactor Netty level based on request type:

Path	Port	CONNECT_TIMEOUT_MILLIS	Rationale
Metadata (GW V1)	443	45s (unchanged)	Account init, address resolution — long timeout is expected
Data plane (GW V2)	10250	5s (new default)	Thin client proxy — fail fast, let SDK retry/failover

The timeout is applied per-request via Reactor Netty's immutable HttpClient.option(), which returns a new config snapshot without mutating the shared client.

---

Diagnostic proof — connectTimeout_Bifurcation_DelayBased

Both ports receive the same 7s SYN-only delay via Linux tc netem + iptables mangle. The only difference is CONNECT_TIMEOUT_MILLIS.

Full CosmosDiagnostics — data plane failure:

{"userAgent":"azsdk-java-cosmos/4.79.0-beta.1 Linux/6.6.87.2-microsoft-standard-WSL2 JRE/21.0.10|F14","activityId":"4b4dbffd-e1ad-43bd-b4c7-9a8f41de6ada","requestLatencyInMs":29998,"requestStartTimeUTC":"2026-03-03T23:24:30.353063100Z","requestEndTimeUTC":"2026-03-03T23:25:00.352036291Z","responseStatisticsList":[],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{},"regionsContacted":["central us","east us 2"],"retryContext":{"statusAndSubStatusCodes":[[503,10001]],"retryCount":1,"retryLatency":7320},"metadataDiagnosticsContext":{"metadataDiagnosticList":[{"metaDataName":"CONTAINER_LOOK_UP","startTimeUTC":"2026-03-03T23:24:30.353779798Z","endTimeUTC":"2026-03-03T23:24:37.866011985Z","durationinMS":7512,"activityId":"6faa8b6d-653d-45ee-aae8-28f67b9e01f7","collectionRid":"UYRLAJa2jxQ="},{"metaDataName":"PARTITION_KEY_RANGE_LOOK_UP","startTimeUTC":"2026-03-03T23:24:37.866118644Z","endTimeUTC":"2026-03-03T23:24:37.992319706Z","durationinMS":126}],"empty":false},"serializationDiagnosticsContext":{"serializationDiagnosticsList":null},"gatewayStatisticsList":[{"sessionToken":null,"operationType":"Read","resourceType":"Document","statusCode":503,"subStatusCode":10001,"requestCharge":0.0,"requestTimeline":[{"eventName":"connectionAcquired","startTimeUTC":"2026-03-03T23:24:37.991135806Z","durationInMilliSecs":5028.05566}],"partitionKeyRangeId":null,"responsePayloadSizeInBytes":0,"httpNetworkResponseTimeout":"PT6S","exceptionMessage":"connection timed out after 5000 ms: thin-client-multi-writer-eastus2.documents.azure.com/40.84.77.67:10250","exceptionResponseHeaders":"{x-ms-substatus=10001}","endpoint":"https://x-eastus2.documents.azure.com:10250/...","e2ePolicyCfg":"{e2eto=PT30S, as=}"},{"sessionToken":null,"operationType":"Read","resourceType":"Document","statusCode":503,"subStatusCode":10001,"requestCharge":0.0,"requestTimeline":[{"eventName":"connectionAcquired","startTimeUTC":"2026-03-03T23:24:43.019820747Z","durationInMilliSecs":5004.424708}],"partitionKeyRangeId":null,"responsePayloadSizeInBytes":0,"httpNetworkResponseTimeout":"PT6S","exceptionMessage":"connection timed out after 5000 ms: thin-client-multi-writer-eastus2.documents.azure.com/40.84.77.67:10250","exceptionResponseHeaders":"{x-ms-substatus=10001}","endpoint":"https://x-eastus2.documents.azure.com:10250/...","e2ePolicyCfg":"{e2eto=PT30S, as=}"},{"sessionToken":null,"operationType":"Read","resourceType":"Document","statusCode":503,"subStatusCode":10001,"requestCharge":0.0,"requestTimeline":[{"eventName":"connectionAcquired","startTimeUTC":"2026-03-03T23:24:48.024689171Z","durationInMilliSecs":5005.594693}],"partitionKeyRangeId":null,"responsePayloadSizeInBytes":0,"httpNetworkResponseTimeout":"PT10S","exceptionMessage":"connection timed out after 5000 ms: thin-client-multi-writer-eastus2.documents.azure.com/40.84.77.67:10250","exceptionResponseHeaders":"{x-ms-substatus=10001}","endpoint":"https://x-eastus2.documents.azure.com:10250/...","e2ePolicyCfg":"{e2eto=PT30S, as=}"},{"sessionToken":null,"operationType":"Read","resourceType":"Document","statusCode":503,"subStatusCode":10001,"requestCharge":0.0,"requestTimeline":[{"eventName":"connectionAcquired","startTimeUTC":"2026-03-03T23:24:54.043417607Z","durationInMilliSecs":5228.884334}],"partitionKeyRangeId":null,"responsePayloadSizeInBytes":0,"httpNetworkResponseTimeout":"PT6S","exceptionMessage":"connection timed out after 5000 ms: thin-client-multi-writer-centralus.documents.azure.com/20.15.133.49:10250","exceptionResponseHeaders":"{x-ms-substatus=10001}","endpoint":"https://x-centralus.documents.azure.com:10250/...","e2ePolicyCfg":"{e2eto=PT30S, as=}"},{"sessionToken":null,"operationType":"Read","resourceType":"Document","statusCode":408,"subStatusCode":20008,"requestCharge":0.0,"requestTimeline":[{"eventName":"connectionAcquired","startTimeUTC":"2026-03-03T23:24:59.273240394Z","durationInMilliSecs":1078.506504}],"partitionKeyRangeId":null,"responsePayloadSizeInBytes":0,"httpNetworkResponseTimeout":"PT6S","exceptionMessage":"Request cancelled by client after reaching timeout specified in end-end timeout policy","exceptionResponseHeaders":"{x-ms-substatus=20008}","endpoint":"","e2ePolicyCfg":"{e2eto=PT30S, as=}"}],"samplingRateSnapshot":1.0,"bloomFilterInsertionCountSnapshot":0,"systemInformation":{"usedMemory":"50496 KB","availableMemory":"2046656 KB","availableProcessors":8},"clientCfgs":{"id":9,"machineId":"uuid:2560edc9-6d4d-4e52-87b3-979d8e765c3f","connectionMode":"GATEWAY","numberOfClients":1,"connCfg":{"rntbd":null,"gw":"(cps:1000, nrto:PT1M, icto:PT1M, cto:PT45S, gwV2Cto:PT5S, p:false, http2:(enabled:true, maxc:10, minc:1, maxs:30))","other":"(ed: true, cs: false, rv: true)"}}}

Reading the diagnostic:

• CONTAINER_LOOK_UP: 7512ms — metadata on port 443 took 7.5s (absorbed 7s SYN delay, succeeded)
• gatewayStatisticsList[0]: connectionAcquired: 5028ms, "connection timed out after 5000 ms: ...eastus2...:10250" — 5s timeout fired
• gatewayStatisticsList[1]: connectionAcquired: 5004ms, same connection timed out after 5000 ms — second attempt, same 5s
• gatewayStatisticsList[2]: connectionAcquired: 5005ms, same — third attempt on eastus2
• gatewayStatisticsList[3]: connectionAcquired: 5228ms, "...centralus...:10250" — failover to Central US, still 5s timeout
• gatewayStatisticsList[4]: 408/20008 — e2e timeout (30s budget) cancelled the fifth attempt
• connCfg.gw: cto:PT45S, gwV2Cto:PT5S — both timeouts visible

The bifurcation proof:

connCfg.gw: (cps:1000, nrto:PT1M, icto:PT1M, cto:PT45S, gwV2Cto:PT5S, p:false, http2:(enabled:true, maxc:10, minc:1, maxs:30))
                                                 ^^^^^^^^  ^^^^^^^^^^^
                                                 metadata   data plane
                                                 (GW V1)    (GW V2)

Port	SYN delay	Timeout config	Outcome	Evidence
443	7s	cto:PT45S	Connected in 7430ms	buildAsyncClient() succeeded
10250	7s	gwV2Cto:PT5S	Timed out at ~5005ms per attempt	connection timed out after 5000 ms

Same network condition. Same delay. Different timeouts. Different outcomes.

---

CosmosDiagnostics — what changes for customers

The clientCfgs.connCfg.gw diagnostic string gains the new gwV2Cto field:

Before:

"gw":"(cps:1000, nrto:PT1M, icto:PT1M, cto:PT45S, p:false, http2:(enabled:true, maxc:10, minc:1, maxs:30))"

After:

"gw":"(cps:1000, nrto:PT1M, icto:PT1M, cto:PT45S, gwV2Cto:PT5S, p:false, http2:(enabled:true, maxc:10, minc:1, maxs:30))"

When data-plane connect times out, each gatewayStatisticsList entry shows:

{
  "statusCode": 503,
  "subStatusCode": 10001,
  "requestTimeline": [{"eventName": "connectionAcquired", "durationInMilliSecs": 5028.05566}],
  "exceptionMessage": "connection timed out after 5000 ms: thin-client-multi-writer-eastus2.documents.azure.com/40.84.77.67:10250",
  "exceptionResponseHeaders": "{x-ms-substatus=10001}",
  "endpoint": "https://thin-client-multi-writer-eastus2.documents.azure.com:10250/..."
}

---

Production code changes

File	Change
Configs.java	New system property COSMOS.THINCLIENT_CONNECTION_TIMEOUT_IN_SECONDS (env: COSMOS_THINCLIENT_CONNECTION_TIMEOUT_IN_SECONDS), default 5s. New method getThinClientConnectionTimeoutInSeconds().
HttpRequest.java	New isThinClientRequest flag + fluent withThinClientRequest(boolean) setter.
ThinClientStoreModel.java	Calls .withThinClientRequest(true) on thin client path requests.
ReactorNettyClient.java	New resolveConnectTimeoutMs(HttpRequest) — applies per-request via .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeoutMs).
HttpClientConfig.java	toDiagnosticsString() emits gwV2Cto alongside cto.

Testing

Test	Technique	Result
connectTimeout_GwV2_DataPlane_1sFiresOnDroppedSyn	iptables -j DROP SYN on :10250	PASSED
connectTimeout_GwV1_Metadata_UnaffectedByGwV2Drop	iptables -j DROP SYN on :10250 only	PASSED
connectTimeout_GwV2_PreciseTiming	iptables -j DROP SYN, 12s e2e	PASSED (3 entries in 12s — proves ~5s per attempt)
connectTimeout_Bifurcation_DelayBased_...	tc netem SYN-only 7s delay on both ports	PASSED (see diagnostic proof above)
ConfigsTests	Unit test for config parsing	PASSED

Test infra: Added manual-thinclient-network-delay to @BeforeSuite/@AfterSuite in TestSuiteBase.java.

Configuration

Property	Env Variable	Default
COSMOS.THINCLIENT_CONNECTION_TIMEOUT_IN_SECONDS	COSMOS_THINCLIENT_CONNECTION_TIMEOUT_IN_SECONDS	5

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

closes #48092

[Copilot]

Pull request overview

This PR improves Azure Cosmos DB Gateway mode behavior when “thin client” (Gateway V2, port 10250) is enabled by applying a shorter per-request TCP connect timeout for data-plane requests, while keeping the existing longer timeout for Gateway V1 metadata requests (port 443). It also expands diagnostics to surface HTTP/2 channel identity and request timeout details, and adds targeted tests (including manual network-manipulation suites) to validate the behavior.

Changes:

• Apply a per-request CONNECT_TIMEOUT_MILLIS in ReactorNettyClient based on whether the request targets the thin client proxy.
• Introduce thin-client-specific timeout policy wiring and new config plumbing (COSMOS.THINCLIENT_CONNECTION_TIMEOUT_IN_SECONDS) plus diagnostics output updates.
• Add/extend unit and fault-injection/manual tests and accompanying docs to validate connect-timeout bifurcation and HTTP/2 connection lifecycle behavior.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/ResponseTimeoutAndDelays.java	Adds Duration-based delay representation alongside seconds.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/ReactorNettyRequestRecord.java	Captures HTTP/2/channel identifiers for richer diagnostics.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/ReactorNettyClient.java	Implements per-request connect timeout selection; captures channel IDs via connection observer.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/HttpTimeoutPolicyForGatewayV2.java	Adds Gateway V2 timeout policy class for thin client document operations.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/HttpTimeoutPolicy.java	Routes eligible thin-client document operations to Gateway V2 timeout policies.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/HttpRequest.java	Adds isThinClientRequest flag + fluent setter for transport customization.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/HttpClientConfig.java	Emits gwV2Cto in diagnostics when thin client is enabled.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/WebExceptionRetryPolicy.java	Switches retry backoff handling to use Duration from timeout policy.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/ThinClientStoreModel.java	Marks thin-client requests and hardens ByteBuf handling for released buffers.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxGatewayStoreModel.java	Ensures request record is available on success/error; sets request URI on cancellation for diagnostics.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentServiceRequest.java	Adds useThinClientMode flag and ensures it is preserved during cloning.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java	Sets useThinClientMode when routing to thin client store model.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/DocumentServiceRequestContext.java	Stores per-attempt ReactorNettyRequestRecord for diagnostics enrichment.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java	Introduces thin client connect-timeout config (sysprop/env) with default value.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/ClientSideRequestStatistics.java	Adds HTTP response timeout, channel IDs, HTTP/2 flag, and e2e policy config to gateway stats serialization.
sdk/cosmos/azure-cosmos/CHANGELOG.md	Documents the new Gateway V2 connect-timeout behavior and timeout policies.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/TestSuiteBase.java	Minor import/format cleanup.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/WebExceptionRetryPolicyTest.java	Extends test coverage for thin-client timeout policies and write behavior.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ConfigsTests.java	Adds unit tests for thin client timeout config parsing and request flag defaulting.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/faultinjection/Http2ConnectionLifecycleTests.java	Adds manual tc netem tests to validate HTTP/2 parent connection survival across real delays/timeouts.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/faultinjection/Http2ConnectTimeoutBifurcationTests.java	Adds manual iptables/tc tests to validate connect-timeout bifurcation by port/path.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/faultinjection/FaultInjectionServerErrorRuleOnGatewayV2Tests.java	Updates thin-client FI tests to account for new Gateway V2 timeout behavior.
sdk/cosmos/azure-cosmos-tests/NETWORK_DELAY_TESTING_README.md	Documents how to run the new manual network-delay lifecycle tests.
sdk/cosmos/azure-cosmos-tests/CONNECT_TIMEOUT_TESTING_README.md	Documents how to run the new manual connect-timeout bifurcation tests.

Comments suppressed due to low confidence (1)

sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/faultinjection/Http2ConnectTimeoutBifurcationTests.java:300

• Method name says “1sFiresOnDroppedSyn”, but the test description and assertions are written for a 5s default connect timeout. Rename the test to match the actual expected behavior (or adjust the timeout setup) so the name stays accurate over time.

    @Test(groups = {TEST_GROUP}, timeOut = TEST_TIMEOUT)
    public void connectTimeout_GwV2_DataPlane_1sFiresOnDroppedSyn() throws Exception {
        // Close and recreate client to ensure no pooled connections exist —
        // we need to force a NEW TCP connection which will hit the iptables DROP.

[FabianMeiswinkel]

LGTM - Thanks!

[xinlian12]

potential a common method:

safeRelease(content.refCnt):
if content.refCnt > 0 -> ReferenceCountUtil.safeRelease(content);

[jeet1995]

Fixed.

[FabianMeiswinkel]

Hmm - I would prefer keeping teh refCnt check explicit instead o fshuffling it into some safeRelase or safeSilentRelease - IMO makes the code flow more readable

[xinlian12]

LGTM, thanks~

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).