Parallel remeshing

[fsalmon001]

Quick_user_guide.odt
Integration of parallel remeshing into nextsim using parmmg2d. The parallel interpolation is also added.

The use of BAMG is still possible.

There will be some minor differences with the develop branch even with bamg due to changes in the explicitSolve function (induce only machine precision errors).

To use parmmg2d, you must compile parmmg2d. It should be done by adding in the container:

wget https://github.com/MmgTools/parMmg2D/archive/refs/heads/nextsim.tar.gz
tar -xzf nextsim.tar.gz
cd parMmg2D-nextsim
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/opt/nextsim_gnu/parmmg2d/
make -j
make install -j

I will edit this message and add a quick user guide for using the parallel remeshing this afternoon.

[tdcwilliams]

thanks @fsalmon001. I have compiled parmmg2d now, so waiting to hear how to run with it

[fsalmon001]

Hi @tdcwilliams, it is in the attached file. Actually, with default options; you just need to add regrid=mmg in the numerics options

[tdcwilliams]

Hi again @fsalmon001, I was able to compile nextsim after export USE_MMG=1
Do I just have to do numerics.regrid=mmg in the config file?

[fsalmon001]

Hi @tdcwilliams,

the USE_MMG=true is done in the environment file like env_compile_gnu_linux.bash. Indeed, with the container, you don't use them so you should add it in the container.

Then, yes, you only need to use numerics.regrid=mmg or do not choose it, as when USE_MMG=true, MMG is the default choice in the options.cpp file (you can change it if you want).

[tdcwilliams]

actually USE_MMG was unset, and it doesn't compile. Where is libmetis?

I have in /opt/local/parmmg2d:

ls /opt/local/parmmg2d/*
/opt/local/parmmg2d/bin:
mmg2d_O3  mmg3d_O3  mmgs_O3  parmmg2d_O3

/opt/local/parmmg2d/include:
mmg  parmmg2d

/opt/local/parmmg2d/lib:
cmake  libmmg.a  libmmg2d.a  libmmg3d.a  libmmgs.a  libparmmg2d.a

/opt/local/parmmg2d/share:
man

and

ls /opt/local/parmmg2d/include/*
/opt/local/parmmg2d/include/mmg:
common	libmmg.h  libmmgf.h  mmg2d  mmg3d  mmgs

/opt/local/parmmg2d/include/parmmg2d:
git_log_pmmg2d.h  libparmmg2d.h  libparmmg2df.h  libparmmg2dtypes.h  libparmmg2dtypesf.h  pmmg2dversion.h

[tdcwilliams]

Hi @fsalmon001,
I guess make install doesn't get metis.
I changed the install directory to be the same as the folder in the tar file, and changed the makefile a bit to get it to compile.
I'll see if I can run it after lunch.

[tdcwilliams]

Hi @fsalmon001,
a short test ran fine with the mmg regridding - I'll see how it goes compared to bamg

[fsalmon001]

Hi @tdcwilliams,

I just pushed a new commit for parmmg2d to have the correct installation path, which was not the case. Now the metis library is at the same location as the parmmg2d library

[tdcwilliams]

Hi @fsalmon001
I also changed CMakeLists.txt to do this with the code below.
It also copies the metis header files to the install location.

CMAKE_MINIMUM_REQUIRED(VERSION 2.8.0)

INCLUDE(CMakeDependentOption)
INCLUDE(CheckCSourceCompiles)
INCLUDE(cmake/macros.cmake)

PROJECT(parmmg2d)

# Must use GNUInstallDirs to install libraries into correct
# locations on all platforms.
include(GNUInstallDirs)

###############################################################################
#####
#####         Release version and date
#####
###############################################################################
SET (CMAKE_RELEASE_VERSION_MAJOR "1")
SET (CMAKE_RELEASE_VERSION_MINOR "0")
SET (CMAKE_RELEASE_VERSION_PATCH "0")
SET (CMAKE_RELEASE_DATE "May 05, 2025")

SET (CMAKE_RELEASE_VERSION
  "${CMAKE_RELEASE_VERSION_MAJOR}.${CMAKE_RELEASE_VERSION_MINOR}.${CMAKE_RELEASE_VERSION_PATCH}")

SET(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)

###############################################################################
#####
#####         Main CMake compilation variables
#####
###############################################################################

SET(PMMG2D_BINARY_DIR ${PROJECT_BINARY_DIR}/src/parmmg2d)
SET(PMMG2D_SHRT_INCLUDE parmmg2d)
SET(PMMG2D_INCLUDE ${PROJECT_BINARY_DIR}/include/pmmg2d)

FILE(MAKE_DIRECTORY ${PMMG2D_BINARY_DIR})

# Create pmmgversion.h file with the release infos
CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/src/pmmg2dversion.h.in
  ${PMMG2D_BINARY_DIR}/pmmg2dversion.h @ONLY)

# Executable path
SET(EXECUTABLE_OUTPUT_PATH ${PROJECT_BINARY_DIR}/bin)
SET(PMMG2D_SOURCE_DIR ${PROJECT_SOURCE_DIR}/src)
#LIST(APPEND CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake/modules)

# Find compilers
IF (${CMAKE_C_COMPILER} MATCHES Clang OR ${CMAKE_C_COMPILER} MATCHES clang)
  # using clang
  SET(CMAKE_COMPILER_IS_CLANG TRUE)
ELSEIF(${CMAKE_C_COMPILER} MATCHES Icc OR ${CMAKE_C_COMPILER} MATCHES icc)
  # using icc
  SET(CMAKE_COMPILER_IS_INTEL TRUE)
ENDIF()

# Hide some options
MARK_AS_ADVANCED(CMAKE_OSX_ARCHITECTURES
  CMAKE_OSX_DEPLOYMENT_TARGET
  CMAKE_OSX_SYSROOT)

IF(CMAKE_COMPILER_IS_GNUCC)
  IF(APPLE)
    # Add flags to the compiler to work on old mac
    ADD_DEFINITIONS( -mmacosx-version-min=10.4 -arch x86_64)

    # To avoid pbs with binary files...
    SET(CMAKE_EXE_LINKER_FLAGS "-arch x86_64 ${CMAKE_EXE_LINKER_FLAGS} ")

    # Determine if the processor supports 64bit execution
    EXECUTE_PROCESS(
      COMMAND sysctl hw.cpu64bit_capable
      ERROR_QUIET
      OUTPUT_VARIABLE 64_CMD
      OUTPUT_STRIP_TRAILING_WHITESPACE
      )
    STRING(REGEX REPLACE "^hw.cpu64bit_capable: (.*)" "\\1" 64_BIT "${64_CMD}")
    #  ELSEIF(UNIX)# UNIX must be after APPLE becaus APPLE is UNIX too
  ENDIF()
ENDIF()

IF(NOT CMAKE_COMPILER_IS_CLANG)
  # Compiler options for profiling... but not possible with clang
  OPTION ( PROFILING "Enable/Disable PROFILING" OFF )
  IF(PROFILING)
    ADD_DEFINITIONS(-pg)
    SET(CMAKE_EXE_LINKER_FLAGS "-pg ${CMAKE_EXE_LINKER_FLAGS}")
  ENDIF(PROFILING)
ENDIF(NOT CMAKE_COMPILER_IS_CLANG)

###############################################################################
#####
#####         Choose executable target to compile
#####
###############################################################################
IF(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
  MESSAGE(STATUS "Setting build type to 'Release' as none was specified.")
  seT(CMAKE_BUILD_TYPE Release CACHE STRING "Choose the type of build." FORCE)
  # Set the possible values of build type for cmake-gui
  SET_PROPERTY(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release"
    "MinSizeRel" "RelWithDebInfo")
  SET(CMAKE_CONFIGURATION_TYPES ${CMAKE_BUILD_TYPE} )
ENDIF()

OPTION ( LIBPARMMG2D_STATIC "Compile static library" ON)
OPTION ( LIBPARMMG2D_SHARED "Compile dynamic library" OFF)

# Explicitly set the DNDEBUG flag in case the user or a parent project overrides
# it.
IF (NOT CMAKE_BUILD_TYPE MATCHES Debug)
    add_definitions(-DNDEBUG)
ENDIF()

############################################################################
#####
#####         MPI
#####
############################################################################
# compile parallel version
ENABLE_LANGUAGE ( Fortran )
FIND_PACKAGE( MPI COMPONENTS C CXX OPTIONAL_COMPONENTS Fortran )

SET( CMAKE_C_COMPILE_FLAGS "${CMAKE_C_COMPILE_FLAGS} ${MPI_COMPILE_FLAGS}" )
SET( CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} ${MPI_LINK_FLAGS}" )

SET( CMAKE_CXX_COMPILE_FLAGS "${CMAKE_CXX_COMPILE_FLAGS} ${MPI_COMPILE_FLAGS}" )
SET( CMAKE_CXX_LINK_FLAGS "${CMAKE_CXX_LINK_FLAGS} ${MPI_LINK_FLAGS}" )

INCLUDE_DIRECTORIES( ${MPI_INCLUDE_PATH} )

# openmpi is bugged (deadlocks) : warn the user
EXECUTE_PROCESS ( COMMAND grep "OMPI_MAJOR_VERSION"  "${MPI_C_INCLUDE_PATH}/mpi.h"
  RESULT_VARIABLE CMD_ERROR
  OUTPUT_VARIABLE CMD_OUTPUT )

IF ( ${CMD_ERROR} MATCHES 0 )
  MESSAGE(WARNING "Possible deadlocks with open-mpi (see https://github.com/open-mpi/ompi/issues/6568 )...")
ENDIF()

SET( CMAKE_C_FLAGS "-DUSE_MPI ${CMAKE_C_FLAGS}" )
SET( CMAKE_CXX_FLAGS "-DUSE_MPI ${CMAKE_CXX_FLAGS}" )
MESSAGE( STATUS "Compilation with mpi" )
SET( LIBRARIES ${MPI_C_LIBRARIES} ${LIBRARIES} )
SET( LIBRARIES ${MPI_CXX_LIBRARIES} ${LIBRARIES} )

EXECUTE_PROCESS ( COMMAND  ${MPIEXEC} --help mapping
  RESULT_VARIABLE CMD_ERROR
  OUTPUT_VARIABLE CMD_OUTPUT )

STRING(REGEX REPLACE "\"" " " CMD_OUT "${CMD_OUTPUT}")

IF ( "${CMD_OUT}" MATCHES "oversubscribe" )
  SET ( MPI_ARGS "-oversubscribe" )
ENDIF()

# Check the minimum MPI version
SET ( CMAKE_REQUIRED_INCLUDES "${MPI_INCLUDE_PATH}" )
SET ( CMAKE_REQUIRED_LIBRARIES "${MPI_C_LIBRARIES}" )
CHECK_C_SOURCE_COMPILES( "
#include <stdio.h>
#include <stdint.h>
#include <mpi.h>
int main(int argc, char **argv) {
MPI_Comm comm_shm = 0;
MPI_Comm_split_type( MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL,
  &comm_shm );
}
"
MPI3_TEST_COMPILES )
IF( NOT MPI3_TEST_COMPILES )
  MESSAGE( FATAL_ERROR "ERROR: impossible to compile a simple MPI test program."
    " Check your MPI library version:"
    " ParMmg requires an MPI version >= 3.0.0.")
ENDIF ( )

SET ( COMPILER_CFG -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
     -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
     -DCMAKE_Fortran_COMPILER=${CMAKE_Fortran_COMPILER} )

SET ( FLAGS_CFG -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
  -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS}
  -DCMAKE_Fortran_FLAGS=${CMAKE_Fortran_FLAGS}
  -DCMAKE_EXE_LINKER_FLAGS=${CMAKE_EXE_LINKER_FLAGS} )

############################################################################
#####
#####         Scotch
#####
############################################################################
# Find SCOTCH library?
SET(SCOTCH_DIR "" CACHE PATH "Installation directory for scotch")

# add Scotch library?
OPTION ( USE_SCOTCH "Use SCOTCH TOOL for renumbering" OFF )

IF ( USE_SCOTCH )

  FIND_PACKAGE(SCOTCH)

  IF ( NOT SCOTCH_FOUND )
    MESSAGE ( WARNING "Scotch library not found:"
      "Using scotch reduce the execution time of mmg2d "
      "(see https://gforge.inria.fr/frs/?group_id=248 to download it)."
      "If you have already installed Scotch and want to use it, "
      "please set the CMake variable or environment variable SCOTCH_DIR "
      "to your scotch directory.")
  ELSE ( )
    SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DUSE_SCOTCH")
    MESSAGE(STATUS "Compilation with scotch: ${SCOTCH_LIBRARIES}")
    SET( LIBRARIES ${LIBRARIES} ${SCOTCH_LIBRARIES})
  ENDIF ( )

ENDIF ( )

SET ( SCOTCH_CFG -DUSE_SCOTCH=${USE_SCOTCH} -DSCOTCH_DIR=${SCOTCH_DIR}
     -DSCOTCH_scotch.h_DIRS=${SCOTCH_scotch.h_DIRS}
     -DSCOTCH_scotch_LIBRARY=${SCOTCH_scotch_LIBRARY}
     -DSCOTCH_scotcherrexit_LIBRARY=${SCOTCH_scotcherrexit_LIBRARY} )

# Find PT-SCOTCH library?
SET(PTSCOTCH_DIR "" CACHE PATH "Installation directory for PT-Scotch")

# add Scotch library?
OPTION ( USE_PTSCOTCH "Use PT-SCOTCH TOOL for repartitioning" OFF )

IF ( USE_PTSCOTCH )

  FIND_PACKAGE(PTSCOTCH)

  IF ( NOT PTSCOTCH_FOUND )
    MESSAGE ( WARNING "PT-Scotch library not found:"
      "Using PT-Scotch reduce the final partitioning time of parmmg2d. "
      "If you have already installed PT-Scotch and want to use it, "
      "please set the CMake variable or environment variable PTSCOTCH_DIR "
      "to your scotch directory.")
  ELSE ( )
    SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DUSE_PTSCOTCH")
    MESSAGE(STATUS "Compilation with PT-Scotch: ${PTSCOTCH_LIBRARIES}")
    SET( LIBRARIES ${LIBRARIES} ${PTSCOTCH_LIBRARIES})
  ENDIF ( )

ENDIF ( )

SET ( PTSCOTCH_CFG -DUSE_PTSCOTCH=${USE_PTSCOTCH} -DSCOTCH_DIR=${PTSCOTCH_DIR}
     -DPTSCOTCH_scotch.h_DIRS=${PTSCOTCH_scotch.h_DIRS}
     -DPTSCOTCH_scotch_LIBRARY=${PTSCOTCH_scotch_LIBRARY}
     -DPTSCOTCH_scotcherrexit_LIBRARY=${PTSCOTCH_scotcherrexit_LIBRARY} )


############################################################################
#####
#####         VTK (to parse (p)vtp/(p)vtu files )
#####
############################################################################
OPTION ( USE_VTK "Use VTK I/O" OFF )

IF ( USE_VTK )
  FIND_PACKAGE(VTK QUIET)

  IF ( NOT VTK_FOUND )
    MESSAGE ( WARNING "VTK library not found: vtk I/O will not be available.")
  ELSE ( )
    ENABLE_LANGUAGE ( CXX )
    ADD_DEFINITIONS(-DUSE_VTK)
    MESSAGE ( STATUS "Compilation with VTK: add vtp and vtu I/O." )
    INCLUDE ( ${VTK_USE_FILE} )
    SET( LIBRARIES  ${LIBRARIES} "-lstdc++" ${VTK_LIBRARIES} )
  ENDIF ( )
ENDIF ( )

SET ( VTK_CFG -DUSE_VTK=${USE_VTK} -DVTK_DIR=${VTK_DIR} )

############################################################################
#####
#####        MMG2D (for mesh data structure)
#####
############################################################################
INCLUDE ( ExternalProject )
OPTION ( DOWNLOAD_MMG "Download and build automatically Mmg." ON )

IF ( DOWNLOAD_MMG )
  UNSET(MMG_DIR CACHE)
  UNSET(MMG_BUILDDIR CACHE)

  # Use pointmap
  OPTION ( USE_POINTMAP "Use map for point tracking" OFF )

  EXTERNALPROJECT_ADD ( Mmg
    GIT_REPOSITORY https://github.com/MmgTools/mmg.git
    GIT_TAG nextsim
    INSTALL_COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR> --target install
    CMAKE_ARGS ${MMG_ARGS} -DUSE_ELAS=OFF ${COMPILER_CFG} ${FLAGS_CFG}
    ${SCOTCH_CFG} ${VTK_CFG} -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
    -DBUILD=MMG -DBUILD_SHARED_LIBS=${LIBPARMMG2D_SHARED} -DUSE_POINTMAP=${USE_POINTMAP}
    -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX})

  EXTERNALPROJECT_GET_PROPERTY ( Mmg BINARY_DIR )
  SET (MMG_BINARY_DIR ${BINARY_DIR})
  EXTERNALPROJECT_GET_PROPERTY ( Mmg SOURCE_DIR )
  SET (MMG_SOURCE_DIR ${SOURCE_DIR})

  EXTERNALPROJECT_ADD_STEP ( Mmg reconfigure
    COMMAND ${CMAKE_COMMAND} -E remove ${MMG_BINARY_DIR}/CMakeCache.txt
    DEPENDEES update
    DEPENDERS configure )

  INCLUDE_DIRECTORIES(${MMG_BINARY_DIR}/include)

  IF( LIBPARMMG2D_SHARED )
    SET(MMG2D_LIBRARY ${MMG_BINARY_DIR}/lib/libmmg2d${CMAKE_SHARED_LIBRARY_SUFFIX})
  ELSE()
    # default behaviour is to link static libs
    SET(MMG2D_LIBRARY ${MMG_BINARY_DIR}/lib/libmmg2d${CMAKE_STATIC_LIBRARY_SUFFIX})
  ENDIF()
  MESSAGE(STATUS "Compilation with Mmg: ${MMG2D_LIBRARY}")
  SET(LIBRARIES ${MMG2D_LIBRARY} ${LIBRARIES})

  # Additionnal directories to access the Mmg sources
  INCLUDE_DIRECTORIES(${MMG_BINARY_DIR}/src/common)
  INCLUDE_DIRECTORIES(${MMG_SOURCE_DIR}/src/mmg2d)
  INCLUDE_DIRECTORIES(${MMG_SOURCE_DIR}/src/common)

ELSE ( )

  UNSET ( USE_POINTMAP CACHE )

  SET(MMG_DIR "" CACHE PATH "Installation directory for mmg")
  SET(MMG_BUILDDIR "" CACHE PATH "Build directory for mmg")

  MESSAGE ( STATUS "Manual installation of Mmg: please, specify the MMG_DIR and MMG_BUILDDIR CMake variables" )

  FIND_PACKAGE(MMG)

  IF ( NOT MMG_FOUND )

    MESSAGE ( FATAL_ERROR "ERROR: The installation directory for mmg is required:"
      "(see https://github.com/MmgTools/mmg and  download the branch develop)."
      "If you have already installed Mmg and want to use it, "
      "please set the CMake variable or environment variable MMG_DIR "
      "to your mmg directory and the CMake variable or environment variable"
      " MMG_BUILDDIR "
      "to your mmg build directory.")
  ELSE ( )

    INCLUDE_DIRECTORIES(${MMG_INCLUDE_DIRS})

    MESSAGE(STATUS "Compilation with Mmg: ${MMG_LIBRARIES}")

    SET(LIBRARIES ${MMG_LIBRARIES} ${LIBRARIES})

    # Additionnal directories to access the Mmg sources
    INCLUDE_DIRECTORIES(${MMG_BUILDDIR_INTERNAL}/src/common)
    IF ( MMG_DIR )
      INCLUDE_DIRECTORIES(${MMG_DIR}/src/mmg2d)
      INCLUDE_DIRECTORIES(${MMG_DIR}/src/common)
    ELSE ( )
      MESSAGE ( FATAL_ERROR "ERROR: The source directory for mmg is required:"
        "(see https://github.com/MmgTools/mmg and  download the branch develop)."
        "If you have already installed Mmg and want to use it, "
        "please set the CMake variable or environment variable MMG_DIR "
        "to your mmg directory.")
    ENDIF ( )
  ENDIF ( )
ENDIF ( )

############################################################################
#####
#####                       Metis
#####
############################################################################
OPTION ( DOWNLOAD_METIS "Download and build automatically Metis." ON )

OPTION ( USE_METIS "Use Metis to partition the domain." ON )

IF ( DOWNLOAD_METIS )
  UNSET ( METIS_DIR CACHE )

  # Metis
  EXTERNALPROJECT_ADD ( Metis
    URL ${CMAKE_CURRENT_SOURCE_DIR}/dependencies/metis-5.1.0.tar.gz
    UPDATE_COMMAND ""
    CONFIGURE_COMMAND ${CMAKE_MAKE_PROGRAM} config prefix=../../../Metis-install
    BUILD_IN_SOURCE 1
    BUILD_COMMAND unset MFLAGS && unset MAKEFLAGS && unset MAKELEVEL && ${CMAKE_MAKE_PROGRAM}
    INSTALL_COMMAND unset MFLAGS && unset MAKEFLAGS && unset MAKELEVEL && ${CMAKE_MAKE_PROGRAM} install
    )

  EXTERNALPROJECT_GET_PROPERTY(Metis SOURCE_DIR)
  SET ( METIS_INSTALL_DIR ${SOURCE_DIR}/../Metis-install)

  INCLUDE_DIRECTORIES(${METIS_INSTALL_DIR}/include)
  SET ( METIS_LIBRARIES ${METIS_INSTALL_DIR}/lib/libmetis.a )

  SET(CMAKE_C_FLAGS "-DUSE_METIS ${CMAKE_C_FLAGS}")

  MESSAGE(STATUS
    "Compilation with metis: ${METIS_LIBRARIES}")
  SET( LIBRARIES ${METIS_LIBRARIES} ${LIBRARIES})

  # Install METIS files alongside ParMmg2d
  INSTALL(CODE "
    set(METIS_DIR \"\${CMAKE_CURRENT_BINARY_DIR}/Metis-prefix/src/Metis-install\")
    file(GLOB METIS_HEADERS \"\${METIS_DIR}/include/*\")
    file(GLOB METIS_LIBS \"\${METIS_DIR}/lib/*\")
    if(METIS_HEADERS)
      file(INSTALL DESTINATION \"\${CMAKE_INSTALL_PREFIX}/include\" 
           TYPE FILE FILES \${METIS_HEADERS})
    endif()
    if(METIS_LIBS)
      file(INSTALL DESTINATION \"\${CMAKE_INSTALL_PREFIX}/lib\" 
           TYPE FILE FILES \${METIS_LIBS})
    endif()
  ")

ELSE ( )

  # add metis library?
  SET(METIS_DIR "" CACHE PATH "Installation directory for metis")

  FIND_PACKAGE(METIS)

  IF( METIS_FOUND)
    IF( USE_METIS )
      # Set flags for building test program
      INCLUDE_DIRECTORIES(${METIS_INCLUDE_DIRS})

      SET(CMAKE_C_FLAGS "-DUSE_METIS ${CMAKE_C_FLAGS}")

      MESSAGE(STATUS
        "Compilation with metis: ${METIS_LIBRARIES}")
      SET( LIBRARIES ${METIS_LIBRARIES} ${LIBRARIES})
    ENDIF()
  ELSE ( )
    MESSAGE(STATUS ${METIS_INCLUDE_DIR} ${METIS_LIBRARIES} )

    MESSAGE ( ERROR " Metis library not found: "
      "If you have already installed Metis and want to use it, "
      "please set the CMake variable or environment variable METIS_DIR "
      "to your metis directory.")
  ENDIF ( )

  IF ( NOT ${METIS_INTSIZE} EQUAL 4 )
    MESSAGE ( WARNING "Metis integer uses ${METIS_INTSIZE} bytes."
      " Only int32 are supported." )
  ENDIF ( )

ENDIF ( )

############################################################################
#####
#####        ParMetis
#####
############################################################################

# add parmetis library?
OPTION ( USE_PARMETIS "Use ParMetis to partition the domain." OFF )

IF( USE_PARMETIS )

  SET(PARMETIS_DIR "" CACHE PATH "Installation directory for parmetis")
  FIND_PACKAGE(Parmetis)

  IF (PARMETIS_FOUND)
    # Set flags for building test program
    INCLUDE_DIRECTORIES(${PARMETIS_INCLUDE_DIRS})

    SET(CMAKE_C_FLAGS "-DUSE_PARMETIS ${CMAKE_C_FLAGS}")

    MESSAGE(STATUS
      "Compilation with parmetis: ${PARMETIS_LIBRARIES}")
    SET( LIBRARIES ${PARMETIS_LIBRARIES} ${LIBRARIES})

  ELSE ( )

    MESSAGE(STATUS ${PARMETIS_INCLUDE_DIR} ${PARMETIS_LIBRARY} )
  
    MESSAGE ( ERROR " parmetis library not found: "
      "If you have already installed ParMetis and want to use it, "
      "please set the CMake variable or environment variable PARMETIS_DIR "
      "to your parmetis directory.")
  ENDIF ( )

ENDIF ( )

############################################################################
#####
#####         Fortran header: libparmmgf.h
#####
############################################################################

ADD_EXECUTABLE(genheader_pmmg2d ${PROJECT_SOURCE_DIR}/scripts/genheader.c)

GENERATE_FORTRAN_HEADER ( pmmg2dtypes
  ${PMMG2D_SOURCE_DIR} libparmmg2dtypes.h
  ${PMMG2D_SHRT_INCLUDE}
  ${PMMG2D_BINARY_DIR} libparmmg2dtypesf.h
  )

GENERATE_FORTRAN_HEADER ( pmmg2d
  ${PMMG2D_SOURCE_DIR} libparmmg2d.h
  ${PMMG2D_SHRT_INCLUDE}
  ${PMMG2D_BINARY_DIR} libparmmg2df.h
  )

###############################################################################
#####
#####         Sources and libraries
#####
###############################################################################
# Header files
INCLUDE_DIRECTORIES(${PMMG2D_SOURCE_DIR})

# Source files
FILE(
  GLOB
  pmmg2d_library_files
  ${PMMG2D_SOURCE_DIR}/*.c
  ${PMMG2D_SOURCE_DIR}/*.cpp
  )
LIST(REMOVE_ITEM pmmg2d_library_files
  ${PMMG2D_SOURCE_DIR}/parmmg2d.c
  ${REMOVE_FILE})
FILE(
  GLOB
  pmmg2d_main_file
  ${PMMG2D_SOURCE_DIR}/parmmg2d.c
  )

IF (NOT WIN32)
  FIND_LIBRARY(M_LIB m)
  SET( LIBRARIES ${LIBRARIES} ${M_LIB})
ENDIF()

SET(CMAKE_MACOSX_RPATH 1)

############################################################################
#####
#####         Compile parmmg2d libraries
#####
############################################################################
SET(LIBRARY_OUTPUT_PATH ${CMAKE_BINARY_DIR}/lib)

IF ( LIBPARMMG2D_STATIC )
  ADD_AND_INSTALL_LIBRARY ( lib${PROJECT_NAME}_a STATIC
    "${pmmg2d_library_files}" ${PROJECT_NAME} )

  IF ( DOWNLOAD_MMG )
    Add_Dependencies(lib${PROJECT_NAME}_a Mmg)
  ENDIF ( )

  IF ( DOWNLOAD_METIS )
    Add_Dependencies(lib${PROJECT_NAME}_a Metis)
  ENDIF ( )

ENDIF()

# Compile shared library
IF ( LIBPARMMG2D_SHARED )
  ADD_AND_INSTALL_LIBRARY ( lib${PROJECT_NAME}_so SHARED
    "${pmmg2d_library_files}" ${PROJECT_NAME} )

  IF ( DOWNLOAD_MMG )
    Add_Dependencies(lib${PROJECT_NAME}_so Mmg)
  ENDIF ( )

  IF ( DOWNLOAD_METIS )
    Add_Dependencies(lib${PROJECT_NAME}_so Metis)
  ENDIF ( )

ENDIF()

# parmmg header files needed for library
SET( pmmg2d_headers
  ${PMMG2D_SOURCE_DIR}/libparmmg2d.h
  ${PMMG2D_BINARY_DIR}/libparmmg2df.h
  ${PMMG2D_SOURCE_DIR}/libparmmg2dtypes.h
  ${PMMG2D_BINARY_DIR}/libparmmg2dtypesf.h
  ${PMMG2D_BINARY_DIR}/pmmg2dversion.h
  )
IF (NOT WIN32 OR MINGW)
  LIST(APPEND pmmg2d_headers  ${PMMG2D_BINARY_DIR}/git_log_pmmg2d.h )
ENDIF()

# Install header files in /usr/local or equivalent
INSTALL(FILES ${pmmg2d_headers} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/parmmg2d COMPONENT headers )

# Copy header files in project directory at every build step
COPY_HEADERS_AND_CREATE_TARGET ( ${PMMG2D_SOURCE_DIR} ${PMMG2D_BINARY_DIR} ${PMMG2D_INCLUDE} )

install(EXPORT ParMmg2dTargets
  FILE ParMmg2dTargets.cmake
  NAMESPACE ParMmg2d::
  DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/parmmg2d
  )

IF (LIBPARMMG2D_STATIC OR LIBPARMMG2D_SHARED)
  SET(LIBPARMMG2D_INTERNAL ON )
ELSE()
  SET(LIBPARMMG2D_INTERNAL OFF )
ENDIF()

###############################################################################
#####
#####         Compile PMMG2D executable
#####
###############################################################################
ADD_AND_INSTALL_EXECUTABLE ( ${PROJECT_NAME} "${pmmg2d_library_files}" ${pmmg2d_main_file} )

IF ( DOWNLOAD_MMG )
  Add_Dependencies(parmmg2d Mmg)
ENDIF ( )

IF ( DOWNLOAD_METIS )
  Add_Dependencies(parmmg2d Metis)
ENDIF ( )

############################################################################
#####
#####         Print git hash to source file
#####
############################################################################

IF (NOT WIN32 OR MINGW)

  ADD_CUSTOM_TARGET(GenerateGitHash
    COMMAND ./git_log_pmmg.sh ${PROJECT_SOURCE_DIR} ${PMMG2D_BINARY_DIR}
    WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/scripts/
    COMMENT "Getting git commit hash"
  )
  ADD_DEPENDENCIES(${PROJECT_NAME} GenerateGitHash)
  IF( LIBPARMMG2D_STATIC )
    ADD_DEPENDENCIES(lib${PROJECT_NAME}_a GenerateGitHash)
  ENDIF ()
  IF( LIBPARMMG2D_SHARED )
    ADD_DEPENDENCIES(lib${PROJECT_NAME}_so GenerateGitHash)
  ENDIF ()

  INCLUDE_DIRECTORIES(${PMMG2D_BINARY_DIR})

ENDIF ()

###############################################################################
#####
#####         Create API Documentation
#####
###############################################################################
FIND_PACKAGE(Doxygen)
IF(DOXYGEN_FOUND)

  # PARMMG Documentation
  CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/doc/doxygen/Doxyfile.in
    ${CMAKE_CURRENT_SOURCE_DIR}/doc/doxygen/Doxyfile @ONLY)

  ADD_CUSTOM_TARGET(parmmg2d_doc
    COMMAND ${DOXYGEN_EXECUTABLE}
    ${CMAKE_CURRENT_SOURCE_DIR}/doc/doxygen/Doxyfile
    # COMMAND ${CMAKE_COMMAND} -E chdir
    # ${CMAKE_CURRENT_SOURCE_DIR}/doc/doxygen/latex make
    # COMMAND cp ${CMAKE_CURRENT_SOURCE_DIR}/doc/doxygen/latex/refman.pdf
    # ${CMAKE_CURRENT_SOURCE_DIR}
    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/doc/doxygen/
    COMMENT "Generating PARMMG2D API documentation with Doxygen.
    Open the doc/doxygen/html/index.html file to see it." VERBATIM
    )

  if ( NOT TARGET doc )
    ADD_CUSTOM_TARGET(doc
      DEPENDS parmmg_doc
      COMMENT "Generating PARMMG2D API documentation with Doxygen.
     Open the doc/doxygen/html/index.html file to see it" VERBATIM
      )
  else()
    add_dependencies(doc parmmg2d_doc)
  endif()
ENDIF ( DOXYGEN_FOUND )

[tdcwilliams]

Hi @fsalmon001, Your fix was nicer though and also works.
I'll do some fuller tests tomorrow.

[tdcwilliams]

Hi @fsalmon001
I am getting this error when I run with numerics.regrid=bamg

nput Error: Incorrect ncuts.
terminate called after throwing an instance of 'std::bad_array_new_length'

[fsalmon001]

Hi @tdcwilliams,
Does your commit solve the error? I don't have this issue here, but maybe you have other options. Regarding the coef_min and coef_max, they are factor in front of mesh size, so I don't think a coef of 0 is good for a default value as this prescribed a minimum size of 0 when using mmg. Maybe a coef_min of 0.99 and a coef_max of 1.01 could be a solution to have different values.

[tdcwilliams]

Hi again @fsalmon001,
otherwise MMG seems to be working pretty well.

For large_arctic_10km.msh and 10days, the timings are:
CPUS 32 64 128
MMG 6:32 4:26 4:01
BAMG (from develop) 8:01 6:33 7:01

[tdcwilliams]

Hi @fsalmon001
the commit solved an MMG init error, something like hmin must be strictly lower than hmax. The new default metric_coef_min is 0.9 (not 0), and this works fine. Would 0.99 and 1.01 be better though?

[fsalmon001]

Ok, I see, actually it was not really an error even if MMG complains about it @tdcwilliams

h_min = coef_min * h_mid
h_max = coef_max * h_mid

So to have a strictly uniform mesh, you need h_min = h_max, or at least, h_min almost equal to h_max. So, I would choose 0.999 and 1.001 for instance, to avoid the warning and still have a uniform mesh by default.

You still have the issue with BAMG?

[tdcwilliams]

yes I still have a bamg error, with this cfg file and 32 cpus

[mesh]
filename=large_arctic_10km.msh
partitioner-space=memory
[setup]
ice-type=topaz4_rean
ocean-type=topaz4_rean
atmosphere-type=era5
bathymetry-type=etopo
use_assimilation=false
dynamics-type=bbm
atmospheric_forcing_input_path=/cluster/projects/nn9878k/sim/data/ERA5_DROWNED
oceanic_forcing_input_path=/cluster/projects/nn9878k/sim/data/TOPAZ4_rean_merged_30m/daily_mean
[simul]
spinup_duration=0
timestep=900
time_init=1996-01-02
duration=10
[thermo]
use_assim_flux=false
assim_flux_exponent=4
diffusivity_sss=0
diffusivity_sst=0
ocean_nudge_timeS_days=10
ocean_nudge_timeT_days=10
freezingpoint-type=linear
h_young_max=2.3000000000000e-01
alb_ice=8.2000000000000e-01
alb_sn=9.0000000000000e-01
albedoW=7.0000000000000e-02
melt_type=2
PhiM=2.0000000000000e-01
snow_cond=2.0000000000000e-01
[dynamics]
time_relaxation_damage=15
compression_factor=7.0000000000000e+03
C_lab=1.3500000000000e+06
substeps=120
use_temperature_dependent_healing=true
ECMWF_quad_drag_coef_air=2.0000000000000e-03
quad_drag_coef_water=6.1625000000000e-03
Lemieux_basal_k1=7
compaction_param=-30
oceanic_turning_angle=20
[age]
include_young_ice=false
[restart]
start_from_restart=false
type=extend
write_initial_restart=true
write_interval_restart=true
write_final_restart=false
output_interval=1
check_restart=true
input_path=nextsim_outputs/restart
basename=19960126T000000Z
[output]
output_per_day=0
datetime_in_filename=true
exporter_path=nextsim_outputs
[moorings]
use_moorings=true
spacing=20
output_timestep=2.5000000000000e-01
file_length=monthly
variables=conc
variables=thick
variables=snow
variables=velocity
variables=sialb
variables=albedo
variables=conc_myi
variables=conc_young
variables=age
variables=age_det
variables=ridge_ratio
[numerics]
regrid=bamg
[drifters]
use_osisaf_drifters=true
use_sidfex_drifters=false
sidfex_drifters_output_time_step=1.2500000000000e-01

[fsalmon001]

I tried your case @tdcwilliams, but without the ocean model (constant instead) and the ice-type is piomas. I don't have the drifter file neither so I did not use drifters. And it works.

I think I have only seen this kind of error, and if I remember well, it is related to an issue with a netcdf file (netcdf uses the argument ncuts in some functions).

[tdcwilliams]

Hi @fsalmon001
the traceback seems to have the error somewhere in bamg.
But instead of trying to debug it, I had a word with @einola and we think we should just delete bamg altogether.

Can you do this (NB remove ifdef MMG and USE_MMG), and then I'll test out MMG again?

[fsalmon001]

Hi @tdcwilliams,

I would agree with you, but we still use one interpolation function of BAMG (for nodes) in interpFields_parallel, so we cannot remove completly BAMG so far. And it is not as easy to remove it because this function calls other functions in BAMG.

Moreover, on a supercomputer, I had an error in the node interpolation function, probably due to epsilon error and a point which was outside the domain for maybe 1e-14 m, and this caused a crash. So there is indeed not perfect functions in BAMG.

So what do we do? I could do what you want relatively quickly I think. But maybe, before, you could try 2 km meshes for instance, to check everything is ok even with finer meshes and your configuration file (I don't have as many options as you in my tests).

[einola]

Yes, there is no need for us to have two remeshing methods, even though @fsalmon001 obviously needs to compare the two in his paper.

But if we remove bamg, we need to update the Docker and Aptainer files in https://github.com/nansencenter/nextsim-env before merging into develop.

[tdcwilliams]

Hi @fsalmon001,
OK I will also try to do some more test runs with finer meshes before you remove the bamg option.

Hi @einola,
I have been using apptainer with mmg installed in the image, so I can push that soon; I will also adapt the Dockerfile and make sure it runs with mmg in docker as well, and then get @akorosov to push it to dockerhub.

[fsalmon001]

I just thought of it @tdcwilliams, @einola, but when the paper will be under review, maybe the reviewers should have access to a version with MMG and BAMG to check and compare the results?

Maybe we could let BAMG in this version, and I do another pull request to remove BAMG from the code after? My post-doc ends at the end of January, so we have enough time, it will be a quick PR.

I also had a look on the interpolation function from BAMG, I think I can code another one, but not sure so far that I will be as quick as the original one. I will keep you informed

[tdcwilliams]

Hi @fsalmon001, we can tag the current version of develop, and the one after this PR, if they want to compare bamg and mmg.
We shouldn't really merge this code if the bamg option is unstable.

[fsalmon001]

Hi @tdcwilliams, yes that sounds good.

For BAMG, I am not convinced that the bamg option is unstable in this PR, there is only one difference when using BAMG from the current dev branch, it is in the explicitSolve function, and the difference is epsilon. So, I don't see how this branch could bring issues when using BAMG compared to the current nextsim version.

[tdcwilliams]

Hi @fsalmon001
I tried with large_arctic_5km.msh and got this error with mmg:

terminate called after throwing an instance of 'ErrorException'
  what():  kkk>=1000

same config as before but different mesh and timestep is 450s.

[fsalmon001]

With your compile option, it crashes at the beginning like you. So, I will be able to look for this problem

[fsalmon001]

I have found the error at the beginning inside the interpolation function.

The error is stupid and is just at line 86 : for (int k = 0; k <= n_background_nodes; k++)
It is not <= but <. I don't know if this could also be the error of @einola.

I have found another error in the parallel interpolation function after more than 20 time steps. So I am looking for this before making a commit.

[einola]

Yes I agree with that @einola, but then, at time step XX, the code reloads again the data to impose the new forcing properties and does a new interpolation isn't it? So I don't understand why it does not crash at this moment if it crashes during the initialisation in the interpolation, except if some data are only read initially. But if it works like that, it's a good point.

Yes, I see. Well, the forcing data (winds, currents, etc) are much higher resolution than the initial conditions we had problems with. So that still works fine. I don't expect we'll ever be in a situation where the forcing data is so coarse that we have this problem with those.

[fsalmon001]

Ok @einola. I made a commit to correct buffer overflows (same error found by @tdcwilliams in the drifters).

Please, tell me how it works now (here it seems to run)

[tdcwilliams]

Hi @fsalmon001
I still have the same crash, somewhere in steps 4-6 of compute_optimal_metric. Your config file looks the same as mine. The figures from before are from 1996 by the way, so it will be different.

[fsalmon001]

With your compilation options, it crashes exactly at the line where there is an issue. I think if you just add some std::cerr, it should be possible to see where it crashes. I don't really see where it can crash (I think I have never had crashes here), but there are some sqrt, maybe of negative numbers. This might happen due to epsilon precision.

[tdcwilliams]

I have narrowed the crash down to find_z1_z2, I think in the while loop (I don't have comm in there so haven't got a barrier before the printouts) - can you see anything there?

[fsalmon001]

This makes sense, I made a commit about this loop last week because there was some issues.

So the error is inside:

    while ((this->Nst - area_A*d_min - area_B*d_max) * pow(z2,alpha) - d_max*integral_ > 0)
    {
        k -= 1;
        area_B = sum_area[nb_vertices-1] - sum_area[k];
        z2 = sum_gamma[order_number[k]];
        k1 = find_z1(order_number, sum_gamma, nb_vertices, d_min, d_max, alpha, z2);
        area_A = sum_area[k1];
        integral_ = integral[k] - integral[k1];
    }

I think the error is that k becomes negative because there is no solution. Can you check please?
I have to think how to solve this

[fsalmon001]

Can you replace inside find_z1_z2 by this:

    double integral_ = integral[nb_vertices-2] - integral[k1];
    int k = nb_vertices-1;

    double value = (this->Nst - area_A*d_min - area_B*d_max) * pow(z2,alpha) - d_max*integral_;
    int kmin = 0;
    double value_min = 1.e14;
    while (value > 0)
    {
        k -= 1;
        if (k == -1)
        {
            kmin = -kmin;
            break;
        }
        area_B = sum_area[nb_vertices-1] - sum_area[k];
        z2 = sum_gamma[order_number[k]];
        k1 = find_z1(order_number, sum_gamma, nb_vertices, d_min, d_max, alpha, z2);
        area_A = sum_area[k1];
        integral_ = integral[k] - integral[k1];
        value = (this->Nst - area_A*d_min - area_B*d_max) * pow(z2,alpha) - d_max*integral_;
        if (value < value_min)
        {
            value_min = value;
            kmin = k;
        }
    }

    if (kmin < 0) z2 = sum_gamma[order_number[-kmin]];

    double z1 = z2 * pow(d_max/d_min,-1./alpha);

Also, if it doesn't crash anymore, please tell me if you have an error message "INACCURACIES OR ERROR IN THE METRIC COMPUTATION:" and the values given.

[tdcwilliams]

yes, it did get negative. One question: would initialising with int k = nb_vertices; help, since you do k -= 1; immediately? Perhaps it wouldn't find the solution if k = nb_vertices - 1 was the answer?

Anyway, I'll try your code out

[fsalmon001]

Yes indeed you are right. Even if this is not the problem here, it should be initialized by k = nb_vertices

[tdcwilliams]

ok, what about integral_? is it initialised ok, and update ok if we change the intialisation of k?

[tdcwilliams]

ok, with your code it doesn't crash anymore, but I do get

INACCURACIES OR ERROR IN THE METRIC COMPUTATION: 583356 is not equal to 554498

[fsalmon001]

Ok this is not surprising.

Instead, can you try with this please (the first case with k=np_vertices should be done before the loop, but it was not achieved correctly):

    double z2 = sum_gamma[order_number[nb_vertices-1]];

    double area_B = 0.;
    int k1 = find_z1(order_number, sum_gamma, nb_vertices, d_min, d_max, alpha, z2);
    double area_A = sum_area[k1];
    double integral_ = integral[nb_vertices-1] - integral[k1];
    int k = nb_vertices-1;

    while ((this->Nst - area_A*d_min - area_B*d_max) * pow(z2,alpha) - d_max*integral_ > 1.e-11)
    {
        k -= 1;
        area_B = sum_area[nb_vertices-1] - sum_area[k];
        z2 = sum_gamma[order_number[k]];
        k1 = find_z1(order_number, sum_gamma, nb_vertices, d_min, d_max, alpha, z2);
        area_A = sum_area[k1];
        integral_ = integral[k] - integral[k1];
    }

    double z1 = z2 * pow(d_max/d_min,-1./alpha);

[tdcwilliams]

Hi @fsalmon001,
with this change it crashes again.

[fsalmon001]

Thank you @tdcwilliams. And now, can you check a mixed of both to see if the inaccurracy is stille the same?

PS: I edited the code below at 8h55

    double z2 = sum_gamma[order_number[nb_vertices-1]];

    double area_B = 0.;
    int k1 = find_z1(order_number, sum_gamma, nb_vertices, d_min, d_max, alpha, z2);
    double area_A = sum_area[k1];
    double integral_ = integral[nb_vertices-1] - integral[k1];
    int k = nb_vertices-1;

    double value = (this->Nst - area_A*d_min - area_B*d_max) * pow(z2,alpha) - d_max*integral_;
    int kmin = 0;
    double value_min = value;
    while (value > 0)
    {
        k -= 1;
        if (k == -1)
        {
            kmin = -kmin;
            break;
        }
        area_B = sum_area[nb_vertices-1] - sum_area[k];
        z2 = sum_gamma[order_number[k]];
        k1 = find_z1(order_number, sum_gamma, nb_vertices, d_min, d_max, alpha, z2);
        area_A = sum_area[k1];
        integral_ = integral[k] - integral[k1];
        value = (this->Nst - area_A*d_min - area_B*d_max) * pow(z2,alpha) - d_max*integral_;
        if (value < value_min)
        {
            value_min = value;
            kmin = k;
        }
    }

    if (kmin < 0) z2 = sum_gamma[order_number[-kmin]];

    double z1 = z2 * pow(d_max/d_min,-1./alpha);

[tdcwilliams]

didn't crash but gave
INACCURACIES OR ERROR IN THE METRIC COMPUTATION: 583356 is not equal to 554498.

NB I used the latest edit of the code

[fsalmon001]

Ok, so the solution is not the first k and the issue is that there is no solution. I have run a lot of cases and I never faced this. I don't know how it is possible.

Basically, this means that the number of vertices will increase a bit in the new mesh. So it is not a huge issue. The next regrid step, do you have this message again?

[tdcwilliams]

yes I seem to get it every time there is a regrid

[fsalmon001]

The mesh seems ok? I am not sure the origin of the problem is in the metric computation if this happens everytime. Or this is a very particular case which is permanent here for some reason.
Is it from a restart?

[tdcwilliams]

I don't know about the mesh, but the moorings outputs seem ok.
This is still init from piomas.

[fsalmon001]

Ok, I will try to understand if this is theoretically possible. At least, this is not a critical error, rather an inaccurracy on the wanted metric.

[tdcwilliams]

can I save some fields or something for you to test?

[fsalmon001]

Maybe, give me the log file of your run (with the LOG[DEBUG], etc) please? I could compare with mine to see when both simulations differ whereas we have the same config file. Maybe there is still an error in the code, and depending on the configuration, we have it or not

[fsalmon001]

In the 3 commits, I corrected:

1. a bug in the boundary flags function (out-of-bounds access)
2. The bug when the solution in the metric computation is smaller than the machine precision. Now, the computation keeps going, with the wrong number of vertices, but it is not a critical error, the mesh will be just heavier or lighter if this happens (pretty scarce according to my tests). The default field in the metric computation is also changed, as velocity has been found to be a poor field to use in this context.
3. Correction of the cohesion when the mesh is not uniform.