PoCL: Upstream Julia CI fails

[maleadt]

Hi, we have recently encountered a bunch of failing Julia workflows like this one. They have some things in common, AFAICT it's always Julia 1.10, the job runs on Zen4 CPU (AMD EPYC 9V74) and fail with Invalid instruction at <hex>. I've tried to reproduce locally with AVX-512 machine, but all the tests pass for me, and i can't really spend much time on debugging this. Have you seen this issue ? Is this a Julia 1.10 specific bug related to AVX512 ? Unfortunately i don't think we can limit the CPU in Github-hosted runners, however we could try using a self-hosted runner.

Originally posted by @franz in pocl/pocl#2117 (comment)

[vchuravy]

      From worker 36:	Invalid instruction at 0x7ff5bcb2728d: 0x48, 0x3d, 0x01, 0xf0, 0xff, 0xff, 0x73, 0x01, 0xc3, 0x48, 0x8b, 0x0d, 0x5b, 0xbb, 0x0d
      From worker 36:	
      From worker 36:	[8402] signal (4.2): Illegal instruction
      From worker 36:	in expression starting at none:1
      From worker 36:	Allocations: 27461624 (Pool: 27431986; Big: 29638); GC: 40

Feeding that into https://defuse.ca/online-x86-assembler.htm:

Disassembly

Raw Hex (zero bytes in bold):

483D01F0FFFF7301C3488B0D5BBB0D   

String Literal:

"\x48\x3D\x01\xF0\xFF\xFF\x73\x01\xC3\x48\x8B\x0D\x5B\xBB\x0D"

Array Literal:

{ 0x48, 0x3D, 0x01, 0xF0, 0xFF, 0xFF, 0x73, 0x01, 0xC3, 0x48, 0x8B, 0x0D, 0x5B, 0xBB, 0x0D }
Disassembly:

0:  48 3d 01 f0 ff ff       cmp    rax,0xfffffffffffff001
6:  73 01                   jae    0x9
8:  c3                      ret
9:  48                      rex.W
a:  8b                      .byte 0x8b
b:  0d                      .byte 0xd
c:  5b                      pop    rbx
d:  bb                      .byte 0xbb
e:  0d                      .byte 0xd

[vchuravy]

Other disassemblers I am trying show the same... Wild...

[franz]

It seems my initial assessment was incorrect; it's also happening on Julia 1.12 and Windows MSVC CI runs. The CPU (znver4) and Invalid/Illegal instructions remain the common features. I still can't reproduce locally, but since it also happens on Windows CI, this is most likely not a Julia bug.

[vchuravy]

Thanks for letting us know! The error is still confusing, since the disassembly seems to be valid and IIUC a stack misalignment should not be an invalid instruction

[franz]

FWIW I've now added a workaround in the CI that limits the CPU to AVX2, and it the Julia CI now passes the tests on Znver4 machine.