test not passing for CUDA build

[jvarma-intel]

Is test/test.sh intended for the cpu build only or is it expected to pass for the cuda build as well? With my cuda openfish build, I am passing OMP_NUM_THREADS=1 and leaving the remaining parameters (batch size, state len, timestep etc) untouched but the generated blob files are vastly different from the test data provided. Can you explain how exactly changing parameters like batch size and state len will affect the output sequences generated? I see there are some more comprehensive tests in the test/extensive_test.sh with additional blob data paths and param configs for the cuda build. Is there any way to get access to these samples or recommendations on how I can create my own datasets to test against with different openfish parameters? Thank you

[BonsonW]

Is test/test.sh intended for the cpu build only or is it expected to pass for the cuda build as well? With my cuda openfish build, I am passing OMP_NUM_THREADS=1 and leaving the remaining parameters (batch size, state len, timestep etc) untouched but the generated blob files are vastly different from the test data provided. Can you explain how exactly changing parameters like batch size and state len will affect the output sequences generated? I see there are some more comprehensive tests in the test/extensive_test.sh with additional blob data paths and param configs for the cuda build. Is there any way to get access to these samples or recommendations on how I can create my own datasets to test against with different openfish parameters? Thank you

Yes, test/test.sh is intended for the CPU part of the CI/CD pipeline.

We are currently working on a way to test the GPU builds.

Generating test data for this is a quite complex process, I will take a look and see if I can make an easier way to reproduce and test this stuff.

[hasindu2008]

@jvarma-intel

We just added a script to do the GPU build:

make cuda=1 debug=1
scripts/gpu_quick_run.sh fast

scripts/gpu_quick_run.sh will download a dataset [a bit big: >15GB], extract it and run the GPU build on it.

[jvarma-intel]

Thank you for updating the test suite! What are expected ranges for the avg diffs for each of the models? I.e., what numbers would be alarming/indicative of an incorrect build or perhaps other issues?

[BonsonW]

@jvarma-intel

This is what I got on a V100. It's going to deviate a little bit on different GPUs. Usually when something goes wrong you'll get garbage values, and the algorithm is deterministic, so they shouldn't change from run to run.

# fast
comapring files /data/bonwon/slorado_test_data/openfish-blobs/fast_1000c_bwd_NTC.blob and bwd_NTC.blob
tensor max elem diff by 0.00195312500000000000000000000000, avg diff: 0.000030, tens_len: 106688000, n_diffs: 13849056
comapring files /data/bonwon/slorado_test_data/openfish-blobs/fast_1000c_post_NTC.blob and post_NTC.blob
tensor max elem diff by 0.00024396181106567382812500000000, avg diff: 0.000000, tens_len: 106688000, n_diffs: 96962021
comapring files /data/bonwon/slorado_test_data/openfish-blobs/fast_1000c_qual_data.blob and qual_data.blob
tensor max elem diff by 0.99999773502349853515625000000000, avg diff: 0.025673, tens_len: 6664000, n_diffs: 5350276
comapring files /data/bonwon/slorado_test_data/openfish-blobs/fast_1000c_total_probs.blob and total_probs.blob
tensor max elem diff by 527.99926757812500000000000000000000, avg diff: 0.512259, tens_len: 1666000, n_diffs: 470731

# hac
comapring files /data/bonwon/slorado_test_data/openfish-blobs/hac_400c_bwd_NTC.blob and bwd_NTC.blob
tensor max elem diff by 0.00195312500000000000000000000000, avg diff: 0.000030, tens_len: 170700800, n_diffs: 24721953
comapring files /data/bonwon/slorado_test_data/openfish-blobs/hac_400c_post_NTC.blob and post_NTC.blob
tensor max elem diff by 0.00024402141571044921875000000000, avg diff: 0.000000, tens_len: 170700800, n_diffs: 162347234
comapring files /data/bonwon/slorado_test_data/openfish-blobs/hac_400c_qual_data.blob and qual_data.blob
tensor max elem diff by 0.99999618530273437500000000000000, avg diff: 0.009212, tens_len: 2665600, n_diffs: 2190661
comapring files /data/bonwon/slorado_test_data/openfish-blobs/hac_400c_total_probs.blob and total_probs.blob
tensor max elem diff by 475.00042724609375000000000000000000, avg diff: 0.316534, tens_len: 666400, n_diffs: 186961

# sup
comapring files /data/bonwon/slorado_test_data/openfish-blobs/sup_200c_bwd_NTC.blob and bwd_NTC.blob
tensor max elem diff by 0.00195312500000000000000000000000, avg diff: 0.000019, tens_len: 341401600, n_diffs: 36453889
comapring files /data/bonwon/slorado_test_data/openfish-blobs/sup_200c_post_NTC.blob and post_NTC.blob
tensor max elem diff by 0.00027927756309509277343750000000, avg diff: 0.000000, tens_len: 341401600, n_diffs: 331694665
comapring files /data/bonwon/slorado_test_data/openfish-blobs/sup_200c_qual_data.blob and qual_data.blob
tensor max elem diff by 0.99999892711639404296875000000000, avg diff: 0.004351, tens_len: 1332800, n_diffs: 1148520
comapring files /data/bonwon/slorado_test_data/openfish-blobs/sup_200c_total_probs.blob and total_probs.blob
tensor max elem diff by 73.00001525878906250000000000000000, avg diff: 0.243373, tens_len: 333200, n_diffs: 90946

[hasindu2008]

@BonsonW
Could you also put what you get on NVIDIA A100 and AMD MI250X as some references?

[BonsonW]

@jvarma-intel

reference test results here: https://github.com/warp9seq/openfish/tree/ec554d697f223503861787e531007586ad5aa275/test/ref_test_vals

[jvarma-intel]

So these are the results I'm getting on Intel hardware after attempting a SYCL conversion of openfish using Intel's SYCLomatic tool and running the fast model test. Notice the first two metrics are fairly similar, while the last two are clear problem areas.

Below is an example of the output sequences being generated:

I have a hunch that the SYCL kernels are not being launch properly, as the chunk and tid values are seemingly arbitrary (ex. the chunkIDs should at least be 0-1000)

I wonder if you can spot any clues/hints based on this information on what might be causing this difference in the CUDA to SYCL conversion.

[BonsonW]

@jvarma-intel
bwd_NTC and post_NTC tensors are from the bwd_scan and fwd_post_scan kernels respectively.

The qual_data tensor comes from the compute_qual_data kernel, and total_probs comes from generate_sequence, so you might want to look at the implementation of those two.

I'm not really sure what could be wrong beyond that. Are the results you're getting deterministic?

[BonsonW]

@jvarma-intel bwd_NTC and post_NTC tensors are from the bwd_scan and fwd_post_scan kernels respectively.

The qual_data tensor comes from the compute_qual_data kernel, and total_probs comes from generate_sequence, so you might want to look at the implementation of those two.

I'm not really sure what could be wrong beyond that. Are the results you're getting deterministic?

Both these kernels depend on beam_search as well, but that's harder to test on GPU. I'll let you know if I find a better way of isolating the problem.

[jvarma-intel]

Yes the results are deterministic. I'll take a look at those kernels, starting with beam search. If there's any sanity checks you can think of that I can keep in mind while debugging the kernels, do let me know! Appreciate it, thanks.

[BonsonW]

Yes the results are deterministic. I'll take a look at those kernels, starting with beam search. If there's any sanity checks you can think of that I can keep in mind while debugging the kernels, do let me know! Appreciate it, thanks.

The simplest way would be to just write the output from a correct GPU beamsearch implementation to disk, and then plug it into your SYCL kernels. If they come out as expected, then there's something wrong with the beamsearch implementation.

My bandwidth is a bit full atm, but if you're allowed to show me your code, I'm happy to take a look myself.