What to test in CI builds

[cmacmackin]

I've been working on refactoring the CI process. I've separated different build types into different jobs. These are

• debug: compiles with debug flags and code coverage
• release: compiles with optimisation flags
• install: compiles as part of pip-installation

Those all pass on Linux for both gcc and clang. I've now moved on to trying to build on MacOS and have run into some trouble using gcc (which is not the native compiler). Is this something we care about supporting? It is likely just a configuration issue with the environment, but it might be tricky to debug given that I don't have direct access to a Mac machine.

Additionally, the Linux jobs are taking longer than they should, really. I think this is because there is quite a lot that needs to be installed in order to get the desired compiler version. If we were to switch to the default compiler versions then it would probably be quicker. Is there any particular reason for testing those specific versions (gcc 6 and clang 3.8)?

Finally, what compiler(s) should we be testing against on Windows?

[thomasms]

I think supporting gcc on Mac is something we should test. I for one use gcc on Mac.
If possible I would like to test on all major versions of gcc that support C++14 and clang.

For Windows I guess we can just use mingw or clang. Maybe the former might be better.
Thoughts?

[cmacmackin]

I'll have to talk to some of my colleagues who know more about working on Windows.

I'm struggling to work out what causes the issues when I try building with GCC on mac. Could you take a look at the build log for me and see if anything jumps out? https://travis-ci.com/thomasms/peakingduck/builds/150050411

[thomasms]

Hmm not sure, that looks odd.
It seems this could be an issue with gcc and mac 10.15 headers.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90835 and
https://stackoverflow.com/questions/58144137/gcc-missing-headers-on-macos10-15-with-xcode11

[cmacmackin]

Which versions of Xcode and gcc are you using on your mac?

[thomasms]

On my work machine, Xcode is version 9 (I tend not to update it often due to horrible breaking issues with our internal builds). Gcc is 7, 8, and 9.

[cmacmackin]

Found that GCC works when I use Xcode 10.3 or 11.3. I've managed to get it building on Mac and Linux for a number of different compiler versions: https://travis-ci.com/thomasms/peakingduck/builds/150320123. I've now moved on to the Windows builds.

[thomasms]

Nice. Great work - thanks for sorting this.

[cmacmackin]

Just an update on this. I've gotten everything to build on the Windows CI using mingw, but for some reason pytest absolutely refuses to find the shared library extension file, despite it being on the Python path. (A normal Python module could be imported from the same directory without a problem.) I borrowed a colleague's Windows laptop and was able to build and run successfully on there. I'm not sure what the issue is exactly but will continue to experiment.

[cmacmackin]

After extensive debugging, I have concluded that the Travis Windows environment is broken. The issue wasn't that Python couldn't find PEAKINGDUCK.so (or rather, PEAKINGDUCK.pyd, as it is called in Windows). Rather some of the system libraries which PEAKINGDUCK.pyd linked to couldn't find all of their dependencies. See the discussion on the Travis forum. It appears it may be related to the fact that it is running on Windows Server Edition (rather than Desktop Edition) and certain libraries are therefore missing.

I have created a minimal reproducer for this issue and placed it on GitHub. You can see that the same error occurs when trying to run tests. I'm not sure whether this is something specific to modules produced by PyBind or whether any compiled Python extension module would have this issue on the CI server.

In any case, I don't think it is worth investing any more time trying to get the Windows CI running for peakingduck.