Conversation
wence-
added
breaking
We're going to track completion only by rank, not additionally stratified by partition, so there's no need to have insert_finished take a vector finished partitions.
This simplifies the tracking data structures and will make it easier to implement shuffles that don't cross-talk.
wence-
force-pushed
the
wence/fea/shuffle-single-finish
branch
from
de6ef8c to
eb10693
Compare
madsbk
left a comment
madsbk
left a comment
There was a problem hiding this comment.
First pass, this is great. I really like the simplification.
But I think we should mention somewhere (maybe in the wait_any() docstring) that we previously supported finishing on a per-partition basis. In theory this could enable more concurrency, but in practice it didn’t really matter because the user program logic needs all partitions to be received before it can continue.
It might also be worth adding a reference to this PR there.
Done. |
2 participants
Rather than tracking completion per rank stratified by partition, switch to only tracking completion by rank. This is more preparatory work towards making it possible to reuse op-ids in shuffles once they are locally complete.
There are two changes here:
insert_finishedto no longer accept finished partitionsI can split these apart if that's preferred.