Monsoon issues

[iboutle]

I have a suite (u-dr627) which is setup to run on both our internal systems and Monsoon. It runs fine internally, but fails on Monsoon when initialising the xios context, returning the (unhelpful) error:

Running on nid 1131
Running on xname x1030c4s0b1n1
terminate called after throwing an instance of 'std::runtime_error'
  what():  bind() failed
unexpected EOF in pipe
Starting ATP frontend failed: failed to launch ATP frontend

nide1131: rank 55 died from signal 6

Is anyone able to help / spot what is wrong? Is there something else missing that needs changing between the two systems. Currently all that's different is in the flow.cylc - the Spack environment location and the queues. I'm suspicious that the error might be an issue with the environments being different on the 2 machines (e.g. different xios version?), but don't really know what I'm looking for to tell here!

[iboutle]

Update - I've figured out that the suite works fine if I switch the compiler to gnu on Monsoon - which makes me think there is an issue / something different with the CCE setup on Monsoon vs internally?

[yaswant]

@t00sa

[tinyendian]

I had exactly the same problem - CCE builds of XIOS fail at runtime during initialisation,

terminate called after throwing an instance of 'std::out_of_range'
  what():  basic_string::substr: __pos (which is 18446744073709551615) > this->size() (which is 0)

Looks like the C++ std::basic_string::substr method is being given a nonsensical position argument; this doesn't seem to happen with GNU on Monsoon.

I rebuilt the same XIOS version 2701 that is used in the Spack software stack on Monsoon with debug settings (-O0 optimisation level), and that resolved the problem. I haven't pinned down the exact flag(s) that cause the problem, but I can look into it a bit more if that is useful.

[tinyendian]

The following FCM build setup for XIOS works both in debug and in dev mode (I haven't tried "prod" mode yet),

%CCOMPILER      mpicxx
%FCOMPILER      mpif90
%LINKER         mpicxx
%BASE_CFLAGS
%PROD_CFLAGS    -O1 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -O2
%DEBUG_CFLAGS   -g -O0
%BASE_FFLAGS
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -G2
%DEBUG_FFLAGS   -g -O0
%BASE_INC       -D__NONE__
%BASE_LD        -D__NONE__ -lmpi
%CPP            CC
%FPP            CC -E -P -x c
%MAKE           gmake

So it seems that -O2 does work after all. According to the build logs, the Spack CCE build of XIOS that crashes at runtime uses flags -ansi -w -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11 -g -O2. I'll try again tomorrow if I can reproduce the crash with these flags.

[dennissergeev]

many thanks for investigating this @tinyendian!