Recast route caching as an LPM, make multipath decisions locally #539
Description
As of #499, we have a fairly simple cache in front of calls to next_hop. However, this includes the L4 Hash as part of the lookup key to maintain path diversity between flows, as we route via a chosen underlay device each time. As a result, this is going to have fairly poor scaling characteristics.
See the discussion on that PR: we can vastly reduce the number of entries we need, and cast the lookup as an LPM query. However, to do so we need to:
- Walk and refresh the entire set of route entries linked in an IRE, which will reimplement some logic from illumos,
- Locally decide between multiple routes.
Doing so will allow us to maintain a shared route cache of O(sleds), and/or make use of per-packet multipath if we choose to.