DRIVERS-3239: Add exponential backoff to operation retry loop for server overloaded errors

[baileympearson]

DRIVERS-3239

Overview

This PR adds support for a new class of errors (SystemOverloadedError) to drivers' operation retry logic, as outlined in the design document.

Additionally, it includes a new argument to the MongoDB handshake (also defined in the design document).

Python will be second implementer.
Node implementation: mongodb/node-mongodb-native#4806

Testing

The testing strategy is two-fold:

• Building off of Ezra's work to generate unified tests for retryable handshake errors, this PR generates unified tests to confirm that:

  • operations are retried using the new SystemOverloadedError label
  • operations are retried no more than 5 (current MAX_ATTEMPTS, as defined in the spec) times

• Following Iris's work in DRIVERS-1934: withTransaction API retries too frequently #1851, this PR adds a prose test that ensures drivers apply exponential backoff in the retryability loop.

• Update changelog.

• Test changes in at least one language driver.

• Test these changes against all server versions and topologies (including standalone, replica set, and sharded
  clusters).

[blink1073]

It looks like you also need to bump the schema version:

source/client-backpressure/tests/backpressure-retry-loop.yml invalid
[
  {
    instancePath: '/tests/0/operations/3/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3
source/client-backpressure/tests/backpressure-retry-max-attempts.yml invalid
[
  {
    instancePath: '/tests/0/operations/1/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3source/client-backpressure/tests/backpressure-retry-loop.yml invalid
[
  {
    instancePath: '/tests/0/operations/3/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3
source/client-backpressure/tests/backpressure-retry-max-attempts.yml invalid
[
  {
    instancePath: '/tests/0/operations/1/expectError',
    schemaPath: '#/definitions/expectedError/type',
    keyword: 'type',
    params: { type: 'object' },
    message: 'must be object'
  }
]
 using schema v1.3

[blink1073]

WIP Python implementation: mongodb/mongo-python-driver#2635

[blink1073]

All unified and prose tests are passing in the Python implementation.

Edit: we're still failing one unified test, "client.clientBulkWrite retries using operation loop", investigating...

Edit 2: we're all good now

[jyemin]

I only reviewed the specification changes, not the pseudocode or tests. Those are best reviewed by implementers.

[jyemin]

Not sure how we handle the date... Is there an automation for this?

[baileympearson]

Not that I know of. Usually the spec author fills it out before merging

I'll just leave this thread open to remind myself to add changelog dates before merging once all changes are completed.

[stIncMale]

The last reviewed commit is 3e58204.

[stIncMale]

The last reviewed commit is b2b5a4d.

There is only one trivial outstanding comment from me.

[jmikola]

Request for changelog entries on the retryable specs. LGTM otherwise.