How can I set up inference using multiple GPUs? I found that my input video will cause a memory overflow if it is too long.

[Lucky-Jay]

[wbhu]

Good question. I have only tested the "sequential" offload method to save GPU vRAM, but the cost is speed. Inferencing on multiple GPUs may be a good choice. Maybe someone can have a try on it, here is the information about distributed inference, https://huggingface.co/docs/accelerate/usage_guides/distributed_inference

[Lucky-Jay]

I tried using Accelerate to perform parallel inference, but I found that it was not very friendly to 4000-series GPUs, so I chose to manually split the input video frames.