Evaluation degradation w.r.t runtime and memory usage

[eslambakr]

Dears,

Thank you very much for your amazing work!
I have one question regarding the evaluation performance, I noticed that the evaluation step takes almost the same time as training one epoch, which is weird for me.
Also, I noticed the GPU memory also is reserved with the same amount during both phases, i.e., training and testing.
Thus, I feel like the evaluation loop should be optimized more.
I tried to optimize it but unfortunately, I couldn't find the root cause of this degradation in the runtime performance.

Thanks in advance!