Fixing gRPC channel shutdown

[sophiatev]

This PR addresses an issue where a customer is running into an error when attempting to start a new orchestration:

Previous channel ManagedChannelImpl{logId=57, target=localhost:4001} was not shutdown properly!!!

The stack trace indicates that this occurs when attempting to build a new managedSideCarChannel in the DurableTaskGrpcClient. It appears that the previous managedSidecarChannel was not disposed properly in a call to close(). This can happen since we do nothing if the channel is not successfully terminated via a graceful call to shutdown() within the timeout (5 seconds). It can also happen if an InterruptedException is thrown, since we don't attempt any shutdown in that case.

To fix this, this PR initiates a forceful shutdown if the graceful shutdown fails in the timeframe. It also initiates a forceful shutdown in the case of an InterruptedException (but manually sets the thread's state to "interrupted" so that we do not lose this information).

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes are added to the CHANGELOG.md
• I have added all required tests (Unit tests, E2E tests)

Additional information

Additional PR information

[halspang]

Is there anyway to test or verify that this doesn't leave the system in a bad state? Even a test where we call shutdownNow and then recreate the connection and complete a request?

[sophiatev]

Is there anyway to test or verify that this doesn't leave the system in a bad state? Even a test where we call shutdownNow and then recreate the connection and complete a request?

After spending much too long attempting to make this test, I think we are actually implicitly doing this for every test in the IntegrationTests class. This is because this method is called after each test which does a forceful shutdown on both gRPC channels being used.