Use CUDA 12 for cross-compiling aarch/ppc

[h-vetinari]

Trying to solve #1114

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jakirkham]

Think it is failing due to this logic in conda-forge-ci-setup. Think we need another branch like this one

[jakirkham]

Probably need to wait for CDN to pick up the new conda-forge-ci-setup

[h-vetinari]

I know :)
It already got picked up in https://anaconda.org/conda-forge/conda-forge-ci-setup/files (#DL > 0), but I don't yet have a bullet-proof way to determine if something is through the CDN (short of mamba repoquery search x, but that's not really scalable)

[h-vetinari]

This is looking pretty good! libarrow got built successfully already! 🥳

Great job on the new compiler setup that this works out of the box 💪
@jakirkham @adibbley @isuruf

[h-vetinari]

It seems the suspicion I had in conda-forge/conda-forge-ci-setup-feedstock#250 is still relevant - we need to copy libcuda.so to ${QEMU_LD_PREFIX}/usr/lib/.

import: 'pyarrow.cuda'
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/apache-arrow_1689413945820/test_tmp/run_test.py", line 29, in <module>
    import pyarrow.cuda
  File "/home/conda/feedstock_root/build_artifacts/apache-arrow_1689413945820/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.10/site-packages/pyarrow/cuda.py", line 21, in <module>
    from pyarrow._cuda import (Context, IpcMemHandle, CudaBuffer,
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

[h-vetinari]

@jakirkham @adibbley
What's the intention for how $PREFIX/cuda-compat/libcuda.so.1 from cuda-compat should be found? Looks like we need to add that to the path somewhere, at least during the testing phase?

Despite adding cuda-compat as a test-dep here, this currently fails:

    from pyarrow._cuda import (Context, IpcMemHandle, CudaBuffer,
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

I'm actually wondering if we don't want to make this unconditional somehow - feedstocks shouldn't have to add a test-dependency on cuda-compat (or cuda-driver-dev, like cupy is doing), just to get the tests to run. Of course, we're in a bit of a catch 22 here, as we should be testing the artefacts as packaged, but given that we don't have libcuda.so in CI (well, except the CUDA 11 images), we have to do this in some way.

[h-vetinari]

Looks like we can fix this in #1122. I think I'll keep this open to debug & test cross-compilation with CUDA 12, which we'll have to solve eventually anyway.

[jakirkham]

Is this still needed? Or did we manage to address this elsewhere already?

[h-vetinari]

We'll need it eventually, when the lowest supported CUDA version in conda-forge is 12.0

[jakirkham]

Ok am hoping it is easier to do this now than it was at that time. Maybe there will be an opportunity to simplify things here when that time comes