[Bug] CPU Mem utilization grows with training, when Dataloader num_workers>0 #415
Description
Describe the bug
CPU memory utilization grows with training and finally cause OOM when num_workers of Dataloader greater than 0.
Especially when more datasets are used, this mem growth phenomenon becomes more obvious.
Environment
torch 2.3.0+cu121
Other information
No response