Retried HTTP requests append Range headers ad-infinitum

[polarina]

do-client/client-lite/src/util/http_agent.cpp

Line 82 in 95b29af

auto tempList = curl_slist_append(_requestContext.requestHeaders, rangeHeader.c_str());

We experienced this issue through the deviceupdate-agent. When a TCP stream is interrupted, the agent retries the request with a Range request.

When said range request itself gets interrupted, subsequent retries are sent with multiple Range headers present. The CDN silently honours only the first Range header where the agent gets a different set of data in response than what it expected, ultimately resulting in the corruption of the downloaded file.

GET /northeurope/redacted/redacted/redacted.gpt HTTP/1.1
Host: uuid--redacted.b.nlu.dl.adu.microsoft.com
User-Agent: Microsoft-Delivery-Optimization-Lite/10.0.0.1
Accept: */*
Range: bytes=6732-390070271
Range: bytes=39692-390070271
Range: bytes=71184-390070271
Range: bytes=82236-390070271
Range: bytes=97668-390070271
Range: bytes=120796-390070271
Range: bytes=149368-390070271
Range: bytes=178052-390070271
Range: bytes=208084-390070271
Range: bytes=225369-390070271
Range: bytes=242261-390070271

[shishirb-MSFT]

Thanks for reporting. That is certainly an issue. How often are you encountering TCP stream interruptions in your scenarios?

[shishirb-MSFT]

Also, is it feasible for you to retry the download via deviceupdate-agent when you detect the corruption?

[polarina]

Thanks for reporting. That is certainly an issue. How often are you encountering TCP stream interruptions in your scenarios?

When you run devices on all sorts of on-premise networks, you're bound to encounter shenanigans. I don't know how they've achieved this, however a packet capture indicates that they're consistently injecting a TCP RST on all TCP port 80 connections after several kilobytes have been downloaded.

Also, is it feasible for you to retry the download via deviceupdate-agent when you detect the corruption?

We do have remote access and were able to proceed by updating the devices by-hand, bypassing the deviceupdate-agent. The agent itself detects the hash mismatch (from the update manifest) after "completing" the download. It didn't appear to retry any further afterwards, but that would only have been a futile attempt on that network with this bug present.

[shishirb-MSFT]

Thanks. Can the customer avoid resetting TCP connections in that way, as a workaround, until a fix is available? Just want to set expectations - fixing this might stretch into January.

[polarina]

That would be the ideal solution. These kind of communications aren't always easy though.