[Bug] Retry logic in `ReferSegDataset` may cause data bias and infinite recursion

[HovChen]

👋 Hi,

I noticed a potential issue in the __getitem__ method of the ReferSegDataset.

When an invalid sample is encountered (e.g., when len(refs) == 0 or all_rej is True), the code falls back to return self.__getitem__(0).

This approach can cause severe data bias by over-sampling the item at index 0, and it risks infinite recursion if sample 0 is also invalid.

A more robust solution would be to use a while True loop with a continue statement to discard the invalid sample and randomly select a new one.