Possible segementer deregistration race

[ibaldin]

have been doing some testing using e2sar_perf and we're trying to account for every packet across multiple senders and receivers. Along the way, this leads us to notice every missing packet and attempt to determine where/why it was lost. I think we may have found a source of loss that is due to the way e2sar_perf handles deregistering senders as it exits. I'll write out what I think is happening and we can discuss on Fri.
Here's what we see:

• At the tail of an e2sar_perf run, we see a brief burst of discards happening in the LB due to "drop_blocked_src" meaning we've received packets for which the source IP is not in the allowed senders list.
  Here's what I think is happening (cursory code inspection so please point out any misunderstandings on my part):
• e2sar_perf sets the socket send buffer to a big number (typically)
• e2sar_perf starts it sender threads and keeps a stat/counter for every packet that it has passed to a syscall
  • loops calling addToSendQueue for every event
  • loops until all packets have been enqueued via syscall (io_uring, sendmsg or sendmmsg)
• sender is deregistered from control plane
• process exits
  I believe this can lead to 2 possible problems, 1 of which I think we're hitting now:

1. Lots of packets are on the socket tx queue but not yet tx'd, sender gets deregistered before all of those packets hit the wire, deregister completes but tx is still happening, LB discards packets at the tail of the run
   a. I believe this is happening in the test runs that Yatish is doing now
2. (possible issue?) Lots of packets are on socket tx queue but sender process exits -- what happens to these enqueued packets when the process exits? Are they silently deleted by the kernel?
   a. I don't have any evidence that this is happening so maybe we get lucky and they do get transmitted and Add meson.build to build python bindings to C++ #2 here just folds in to another variant of Meeting notes #1.

[ibaldin]

This is likely related to #130