Update pytorch / cuda version?

I reckon with #76 I am encountering issues as my host system is RHEL based and uses CUDA 12.4 and the container is ubuntu 22 and CUDA 11.7.
Is there a chance to update to a newer pytorch and/or cuda version? I tried to base it on rockylinux, but failed so far.