Resumes for subscriber stream transient failures

[dbolduc]

The subscriber StreamingPull RPC needs a resume loop.

We have one bidi stream (in storage), which required a custom resume loop, so we wrote it in the storage crate.

The plan of record is to write the StreamingPull resume loop in the pubsub crate.

This issue is to track the implementation of the resume loop, with some reasonable resume policy. A non-goal is to define the resume policy APIs and make it configurable.

[coryan]

Does this overlap with #3020 ?

[dbolduc]

Does this overlap with #3020 ?

Yes it overlaps, but they are not necessarily the same.

I updated the issue description to say exactly what I intend this issue to cover. Sorry about leaving out context.