Investigate intermittent regression test failures

[bieryAtFnal]

Here is a sampling of intermittent problems that have been seen recently.

1. The tpstream_writing_test in the 16-Aug-2025 overnight running of the regression tests had a "failed to insert into LB" message in one of its log files.

   • Here is a copy of the message: There were 1 failures to insert data into the latency buffer out of 4887 attempts in the latest monitoring interval.
   • Here is a link to the GitHub CI log: link

2. The example_system_test in the 09-Aug-2025 overnight running of the regression tests had a "tp_input_2000: Unable to push within timeout period" message in one of its log files (link).

   • I've seen this problem on other regression tests.
   • When I get a chance, I'd like to propose some tests with mutex locking in our "queue" code...
   • This was also seen in the 3ru_1df test on 12-Sep-2025 (link)
   • And in the DAPHNE_TPG part of the readout_type_scan_test on 15-Sep-2025 (link)
   • And here
   • And in the 3ru_1df test on 11-Oct-2025 (link)

3. I've seen the number of TriggerRecords produced in the example_system_test occasionally fluctuate slightly too low.

   • That happened in the 08-Aug-2025 overnight running of the regression tests. (link)
   • I saw it in a running of all regression tests using dbt-build --integtest on the fddaq-v5.4.0-rc1 candidate release

4. The DAPHNE_TPG part of the readout_type_scan_test complained about an empty TP fragment in the 04-Sep-2025 overnight running of the regression tests.

   • Here is a copy of the error message: Trigger Primitive fragment for SrcID Trigger_0x00000064 in record (5, 0) has size 72 (outside range [96, 4392])

[bieryAtFnal]

Some progress has been made in understanding the third type of error (number of TRs fluctuating low in the example_system_test) and a candidate fix has been provided for that error (#264), but there is still work to be done on that problem and the other problems, so I'll change the Target Release for this Issue to v5.6.0...