Add option to use precomputed templates#1098
Add option to use precomputed templates#1098GMNGeoffrey wants to merge 1 commit intogoogle-deepmind:mainfrom
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
3e6b9ef to
4a1bc0f
Compare
|
I also have additional changes to use an npz file instead, enable just running the featurization code (e.g. so you can do that on a CPU-only machine first), and break the run with precomputed features out into a separate script so it doesn't try to look for all the db files. Happy to upstream those, but as I noted I wanted to get some feedback on the general idea. I'll sort out the CLA on our side |
4a1bc0f to
4b349e4
Compare
My experience is that template processing takes a huge fraction of the overall runtime. It's also all on CPU, which is a waste if you're using GPU machines. Note that this uses the features.pkl which is already saved by the existing code. For more stability across numpy versions, etc. we could instead use `npz` files with `numpy.save`/`numpy.load` as `features.pkl` just holds a dictionary of numpy arrays. I wanted to first gauge the reaction to this general idea though. See discussion in google-deepmind#895
4b349e4 to
a7a2f98
Compare
|
Alright CLA check is passing. @Augustin-Zidek could you PTAL? |
My experience is that template processing takes a huge fraction of the overall runtime. It's also all on CPU, which is a waste if you're using GPU machines.
Note that this uses the
features.pklwhich is already saved by the existing code. For more stability across numpy versions, etc. we could instead usenpzfiles withnumpy.save/numpy.loadasfeatures.pkljust holds a dictionary of numpy arrays. I wanted to first gauge the reaction to this general idea though.See discussion in #895