[ERROR] client reporting Error CQE, when running gpunetio_verbs_put_bw example on two node.

[foraxe]

[ERROR] client reporting Error CQE, when running gpunetio_verbs_put_bw example on two node.
Hi, everyone.

[Description]
Run commands:
Server:

cd /workspace/gpunetio/examples/gpunetio_verbs_put_bw
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/workspace/gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_bw -g 08:00.0 -d mlx5_bond_0

Client:

cd /workspace/gpunetio/examples/gpunetio_verbs_put_bw
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/workspace/gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_bw -g 08:00.0 -d mlx5_bond_0 -c x.x.x.x

When run on single node in tmux, the example returns Sample finished successfully and give the BW results.

But when run on two node, the client return Error CQE. And i also tried -p 1 mode, same errors.

ps: I am trying this example for solving the issue in NIXL gpunetio backend ai-dynamo/nixl#952

[e-ago]

Looking at ai-dynamo/nixl#952 (comment) it seems you are now able to run those samples between two nodes while NIXL still gives you issues. May I ask you what did you change in these GPUNetIO samples to make them work on your system?

Sorry for the late response, will be more responsive from now on :)

[foraxe]

Hi, @e-ago Appreciate your response.
I uploaded the working version of GPUNetIO. ai-dynamo/nixl#1112
Honestly, I didn't have much time after when i fixed the intra-node run. So the code may not be very clean. But we may find a clue for solving the root problems and make it run for internode.
ps: We did try the OOB version from the community codes, but problem still.