[torch.AcceleratorError] causes entire benchmark to crash

[ai-nikolai]

When one of the kernels produces a bug that affects the GPU. all follow-on computations on the GPU are broken.

E.g.:

torch.AcceleratorError: CUDA error: an illegal memory access was encountered

After that even doing some basic torch calculation causes the same torch error. (Therefore it is not possible to recover from this within the same python process (or so it seems at least)).

Have you encountered this? @simonguozirui do you know how to overcome such an error? (I am wondering whether the eval pipeline needs to be re-written for such a case, e.g. via bash scripts so that the eval pipeline can be in independent python processes?)

[ai-nikolai]

Perhaps the answer can be something similar to this https://github.com/wzzll123/MultiKernelBench/blob/05e3ff4d39953ac7c5368f28d09314b0740aa6f0/evaluation.py#L28 in MultiKernelBench

[ai-nikolai]

If anyone is interested in this issue, tag me, we are working on a fix.