[FEATURE] auto-detect `libcudart.so` path / dynamic dependency detection

[ethangraham2001]

Motivation

eBPF can monitor when a file is opened by a binary. Oracle, for instance, uses this for dependency detection in their cloud service. We could do something like this

def on_file_open(path):
    if path.contains("libcudart.so"):
        attach_uprobes(path)

Basically, automatically detect when a libcudart.so is used by a running binary. This would be particularly useful when running jobs that access difference libcudart.so files, for example PyTorch jobs running inside their own separate conda envs and accessing copies of the runtime dynamic lib. Removes the burden from the user who wants to just run their jobs without having to spawn multiple gpuprobe-daemon instances or checking through their env to find where libcudart.so is.

Proposal

Add an option --autodetect-libcudart that monitors such file-open events. This needs to be an option, as it is pretty expensive to perform a string comparison whenever a file is opened.

This is pretty important for usability imho.

[ethangraham2001]

This functionality can potentially be expanded into a full program that allows for dynamic dependency detection of any running process. This isn't unique to CUDA apps so its scope is maybe larger than what the project originally intended, however it would be useful for seeing how a running CUDA job "branches out" and calls dynamic libs.

[CrackedPoly]

You mean extending the monitored library from current libcudart singleton to duplicated ones or even ones with different versions in the machines?

[ethangraham2001]

You mean extending the monitored library from current libcudart singleton to duplicated ones or even ones with different versions in the machines?

If possible, different versions! However, this will mean a different binary layout so I am unsure if this is feasible. Something that will need to be played around with.

Also worth seeing if auto-detection is worthwhile, because that would require more Linux instrumentation namely for open() syscall which will probably incur some overhead on all running binaries, not just CUDA ones