Failed uxm_relock often followed by initialize in tune block

[ktcrowley]

In schedule 874 on SATp2, beginning at ~2300 on 10/29/2025, a relock call crashed for slot c5s5. The stdout within the nextline service reports that this slot then gets dropped from targeted SMuRF calls, but because the next commands for detector tuning follow a run.initialize() call, this slot gets its detectors tuned despite the readout not being in a usable state.

This issue could apply in general where slots that have a failed det_controller action get repeatedly polled to reset their "target" status the next time a tune block appears in the schedule. However, a slot without relock will never recover without that command being rerun, which doesn't happen every tune block. The data of the problem slot won't be useful for any detector tuning or sky observations, and likely shouldn't be streamed.

Would there be away to make a two-tier target system, one for relocks, and the other for detector tunings? This may not be critical if SATp2 slot issues stabilize first, but may still be useful whenever issues with relock commands crop up.

[BrianJKoopman]

Thanks for opening this issue. When I first wrote the 'drop target' code I didn't really imagine us running run.initialize() regularly throughout the schedule -- so the dropped slot wouldn't be picked back up. We started doing this so folks could recover slots mid-schedule without interrupting. Is it possible to recover mid-schedule with uxm_relock?

It seemed like things were stable after we completed the SMuRF server swap late last week. Is c5s5 stable now?

What we'd need internally to track this is to record the operation that caused us to drop the client. Then when we go to initialize() again, have some way to not pick up clients that were dropped for unrecoverable reasons. Or maybe we have some separate function that is meant to be called throughout the schedule to regain recovered controllers, since initialize() was really meant to get us into a clean initial state. I'll give it some thought if we still have this stability issue.

[kmharrington]

I think we decided on the operations call tonight that we're going to wait and see if things are just much more stable now?

[kmharrington]

An update here is that this happened again for SATp2 yesterday so the error is not completely gone.