JEventSource::Close() is not called when JApplication::Run() terminates due to an exception (including timeout)

[nathanwbrei]

[nathanwbrei]

This one is tricky. One could argue that not Close()ing JEventSources is the correct behavior. However:

• If a user component excepts, JANA2 would simply pause, give JMain or equivalent the chance to handle the exception, possibly adjust the topology or component configuration as needed, and resume. As it turns out, this doesn't happen in practice because (1) the generic jana CLI tool cannot know how to correctly respond to exceptions arising from user code, (2) Components cannot be reconfigured after JApplication::Initialize() is called anyhow, and (3) even with custom EICrecon and halld_recon executables, nobody has presented a single case of an exception in user code that can be recovered from at the JANA2 level. So the whole JException mechanism really just exists to provide extremely detailed error messages and shut down processing.

• This has consequences for hd_root. For example, JEventSource_EVIOpp maintains its own multithreaded block splitter, complete with thread pool and dispatcher. (I don't recommend this, by the way). The thread pool and dispatcher are shut down in the destructor. This logic ought to belong in Close(), because JEventSources are designed so that they can be spun up and down as needed based off of the sequence of input files/urls, and all associated resources should be freed as soon as one spins down. However we can't put it in Close(), because Close() won't be called if an exception occurs. If an exception occurs, the JEventSource_EVIOpp will be destroyed when the topology is shut down, but the dispatcher thread will still be running and accessing member variables, eventually leading to segfault.

• Having been forced to think about this more carefully because I'm writing this down, I realize now that there is no contradiction here. If the processing topology is being shut down, and the event sources are being destroyed, there should be a check as to whether they are still open. If they are, they should be closed. This would save down the data that has been successfully processed so far, which could aid in debugging. If the exception is going to be handled in JMain or some equivalent, it will obviously do so before destroying the topology.