Project broken after changing shard size back to 1GB #4
Description
I realize that the shard size was changed from 300MB back to 1GB. In doing so, it seems the main-wds.py script has broken.
When trying to run ./run single or ./run multi, the check_shards() function looks for 490 shards, while now the number of shards are smaller.
I also tried running the main-wds.py file myself, but got this error:
No such file or directory: './shards/imagenet-train-000488.tar'
I found that the error stems from the way the train shards are being loaded. The number 490 is hardcoaded in the parser arg in the line:-
parser.add_argument('--trainshards', default='./shards/imagenet-train-{000000..000490}.tar', help='path/URL for ImageNet shards',)
I changed that to the number of training shards generated for the 1GB/shard (146 in my case), but this is a stop-gap. A programmatic fix might be a better idea.
If you think it might help, I can fix this in a Pull Request. Let me know, and thanks again for all your amazing work on this!