CI jobs on Wino: Install SQLite + move CI job to scratch

[anderkve]

This issue is just a reminder for me and @ChrisJChang so we don't forget the two things mentioned in this closed PR: #515 (comment)

[anderkve]

Update: I've built SQLite from source on Wino and installed it to /scratch2/shared/sqlite. I'm now testing this with CI jobs on the branch ci-linux_sqlite.

[ahye]

You know you could have just asked me to install sqlite and the other thing on wino :D

[anderkve]

Ah, lol, I keep forgetting that you have root access to wino, @ahye! :D

Oh well, sqlite and patch at least now works on wino. But let me know if you anyway decide to install them globally on wino, in that case I'll remove some lines from the CI job script.

But more importantly, some of the backends needed for the test runs still do not build, due to missing packages. According to the CI build output we need the python packages cython, pandas, matplotlib and pyhf. Can you try to pip install those on wino?

For reference, here's the full CI output from the latest attempt at building GAMBIT + backends for test runs: https://productionresultssa9.blob.core.windows.net/actions-results/1cae4ff6-c31b-43ce-afeb-5a5bcff5bc30/workflow-job-run-438b468e-7cf9-5ba8-ec6a-f0ef784ea62e/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-04T08%3A05%3A23Z&sig=hgHnCQIn3EgBV6WI1PnbNiSc9YKakkCDcfQDvoCm%2Bic%3D&ske=2024-11-04T18%3A08%3A11Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-04T06%3A08%3A11Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-04T07%3A55%3A18Z&sv=2024-08-04

[ahye]

The pythons have been installed. I did get warnings like this though

WARNING: The scripts cygdb, cython and cythonize are installed in '/usr/local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-local

I've also installed patch-2.7.6-16.el9.x86_64

For sqlite, my yum claims that
sqlite-libs.x86_64 3.34.1-7.el9_3 @rhel-9-for-x86_64-baseos-rpms
has been installed. Maybe there was problem finding it? I can see stuff at
/usr/lib64/libsqlite3.so.0
/usr/lib64/libsqlite3.so.0.8.6
but this might be what you installed?

[anderkve]

Thanks a lot!

I'll run the CI test again now with the Python packages installed, and using the system-wide installation of patch. I'll keep using my installation of sqlite, as I think we need both the library and the header files (at least we have two cmake variables SQLite3_INCLUDE_DIR and SQLite3_LIBRARY), and I couldn't find any header files corresponding to the library installation in /usr/lib64.

[anderkve]

Not sure why, but the classy backend fails with something cython-related. Have you seen this before, @ChrisJChang ? Maybe it's just a version thing?

2024-11-05T07:58:20.0428428Z running build_ext
2024-11-05T07:58:21.1421899Z 
2024-11-05T07:58:21.1422727Z Error compiling Cython file:
2024-11-05T07:58:21.1424587Z ------------------------------------------------------------
2024-11-05T07:58:21.1426087Z ...
2024-11-05T07:58:21.1426876Z         return d.items()
2024-11-05T07:58:21.1427913Z     else:
2024-11-05T07:58:21.1428785Z         return d.viewitems()
2024-11-05T07:58:21.1429570Z 
2024-11-05T07:58:21.1429983Z ctypedef np.float_t DTYPE_t
2024-11-05T07:58:21.1431418Z ctypedef np.int_t DTYPE_i
2024-11-05T07:58:21.1432470Z          ^
2024-11-05T07:58:21.1433769Z ------------------------------------------------------------
2024-11-05T07:58:21.1434904Z 
2024-11-05T07:58:21.1435935Z classy.pyx:32:9: 'int_t' is not a type identifier
2024-11-05T07:58:21.3619740Z warning: classy.pyx:373:76: local variable 'errmsg' referenced before assignment
2024-11-05T07:58:21.3623297Z warning: classy.pyx:374:39: local variable 'errmsg' referenced before assignment
2024-11-05T07:58:21.8298284Z Traceback (most recent call last):
2024-11-05T07:58:21.8303412Z Compiling /mn/fys-server1/a1/chrichan/GAMBIT/Runner/actions-runner/_work/gambit/gambit/Backends/installed/classy/3.1.0/python/../python/classy.pyx because it changed.
2024-11-05T07:58:21.8311345Z   File "/mn/fys-server1/a1/chrichan/GAMBIT/Runner/actions-runner/_work/gambit/gambit/Backends/installed/classy/3.1.0/python/autosetup.py", line 49, in <module>
2024-11-05T07:58:21.8315298Z     setup(
2024-11-05T07:58:21.8316742Z   File "/usr/lib64/python3.9/distutils/core.py", line 148, in setup
2024-11-05T07:58:21.8318599Z     dist.run_commands()
2024-11-05T07:58:21.8320442Z   File "/usr/lib64/python3.9/distutils/dist.py", line 966, in run_commands
2024-11-05T07:58:21.8325481Z [1/1] Cythonizing /mn/fys-server1/a1/chrichan/GAMBIT/Runner/actions-runner/_work/gambit/gambit/Backends/installed/classy/3.1.0/python/../python/classy.pyx
2024-11-05T07:58:21.8329347Z     self.run_command(cmd)
2024-11-05T07:58:21.8331178Z   File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
2024-11-05T07:58:21.8333122Z     cmd_obj.run()
2024-11-05T07:58:21.8334781Z   File "/usr/lib64/python3.9/distutils/command/build.py", line 135, in run
2024-11-05T07:58:21.8336810Z     self.run_command(cmd_name)
2024-11-05T07:58:21.8338643Z   File "/usr/lib64/python3.9/distutils/cmd.py", line 313, in run_command
2024-11-05T07:58:21.8340808Z     self.distribution.run_command(command)
2024-11-05T07:58:21.8342881Z   File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
2024-11-05T07:58:21.8344824Z     cmd_obj.run()
2024-11-05T07:58:21.8346537Z   File "/usr/lib64/python3.9/distutils/command/build_ext.py", line 340, in run
2024-11-05T07:58:21.8348617Z     self.build_extensions()
2024-11-05T07:58:21.8350856Z   File "/usr/lib64/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
2024-11-05T07:58:21.8353266Z     self._build_extensions_serial()
2024-11-05T07:58:21.8355750Z   File "/usr/lib64/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
2024-11-05T07:58:21.8358254Z     self.build_extension(ext)
2024-11-05T07:58:21.8361383Z   File "/usr/local/lib64/python3.9/site-packages/Cython/Distutils/build_ext.py", line 130, in build_extension
2024-11-05T07:58:21.8364406Z     new_ext = cythonize(
2024-11-05T07:58:21.8367223Z   File "/usr/local/lib64/python3.9/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
2024-11-05T07:58:21.8369899Z     cythonize_one(*args)
2024-11-05T07:58:21.8372888Z   File "/usr/local/lib64/python3.9/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
2024-11-05T07:58:21.8375712Z     raise CompileError(None, pyx_file)
2024-11-05T07:58:21.8380651Z Cython.Compiler.Errors.CompileError: /mn/fys-server1/a1/chrichan/GAMBIT/Runner/actions-runner/_work/gambit/gambit/Backends/installed/classy/3.1.0/python/../python/classy.pyx
2024-11-05T07:58:22.1937615Z usage: autosetup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
2024-11-05T07:58:22.1940375Z    or: autosetup.py --help [cmd1 cmd2 ...]
2024-11-05T07:58:22.1942056Z    or: autosetup.py --help-commands
2024-11-05T07:58:22.1943574Z    or: autosetup.py cmd --help
2024-11-05T07:58:22.1944422Z 
2024-11-05T07:58:22.1945047Z error: option --user not recognized
2024-11-05T07:58:22.2202259Z make[4]: *** [Makefile:195: classy] Error 1
2024-11-05T07:58:22.2209534Z make[3]: *** [CMakeFiles/classy_3.1.0.dir/build.make:88: classy_3.1.0-prefix/src/classy_3.1.0-stamp/classy_3.1.0-build] Error 2
2024-11-05T07:58:22.2214135Z make[2]: *** [CMakeFiles/Makefile2:14892: CMakeFiles/classy_3.1.0.dir/all] Error 2
2024-11-05T07:58:22.2221366Z make[1]: *** [CMakeFiles/Makefile2:15055: CMakeFiles/classy.dir/rule] Error 2
2024-11-05T07:58:22.2228343Z make: *** [Makefile:6949: classy] Error 2

[ChrisJChang]

I haven't seen this before. In one of the cmake steps, it is saying 'Found Cython3.9.18: 3.0' After saying it found version 3.0.11. Although v3.9 is not a thing for cython (we have one of the more up to date versions).

[anderkve]

I think the "3.9.18" refers to the Python version on Wino. Not sure why it enters the Cython name like that, though -- presumably Cython has been built against that specific Python verison.

Another current problem with the CI tests: With sqlite installed on Wino it looks like the "make contur" step runs OK. But the test run with ColliderBit_CMSSM.yaml still fails on Wino, with the message saying that the contur backend is missing. (See below.)

My guess is that something in the initialisation of contur goes wrong (e.g. some python package fails to import), and as a result GAMBIT regards contur as not connected. @ChrisJChang, I guess one of us (whoever gets time first) should just try to manually clone GAMBIT on Wino and reproduce the "test_runs" part of the CI job, to see if we can understand why contur isn't working.

2024-11-05T11:33:40.3367354Z GAMBIT has exited with fatal exception: GAMBIT error
2024-11-05T11:33:40.3368718Z ERROR: A problem has been raised by the dependency resolver subsystem.
2024-11-05T11:33:40.3370387Z Found no candidates for backend requirements of ColliderBit::Contur_LHC_measurements_from_stream:
2024-11-05T11:33:40.3372202Z {( Contur_Measurements, Contur_output(std::shared_ptr<std::ostringstream>,std::vector<std::string>&))}
2024-11-05T11:33:40.3373399Z from group: none
2024-11-05T11:33:40.3374067Z Note that viable candidates exist but have been disabled:
2024-11-05T11:33:40.3375410Z ORIGIN (VERSION)     FUNCTION                         CAPABILITY                                       TYPE                             STATUS 
2024-11-05T11:33:40.3378017Z Contur (2.1.1)       Contur_LogLike_from_stream       Contur_Measurements                              Contur_output(std::shared_ptr<std::ostringstream>,std::vector<std::string>&) -1     
2024-11-05T11:33:40.3379485Z 
2024-11-05T11:33:40.3379647Z Status flags:
2024-11-05T11:33:40.3380478Z  1: This function is available, but the backend and/or its version are 
2024-11-05T11:33:40.3381755Z     not compatible with all relevant rollcall declarations and YAML rules.
2024-11-05T11:33:40.3383011Z  0: This function is not compatible with any model you are scanning.
2024-11-05T11:33:40.3384217Z -1: The backend that provides this function is missing.
2024-11-05T11:33:40.3385347Z -2: The backend is present, but function is absent or broken.