when use streaming data ,how about read speed when training, just save on local. compare to just use pytorch dataloader

[yja1]

when use streaming data ,how about read speed when training, just save on local. compare to just use pytorch dataloader

[VSehwag]

Though StreamingDataset offers loading from a remote, I recommend loading from local because it's fast. In loading from a remote (e.g., AWS s3 buckets) one would need a large cache to avoid thrashing. When training the large parameter models with descent SSDs and <=8 gpus, the dataloading shouldn't be a bottleneck.