Running setup.py

[eogbe]

I m trying to use the HSL subroutine in python and when i run python setup.py install i get the following error:

Traceback (most recent call last):
File "setup.py", line 77, in
use_cython = hsl_config.getboolean('CODE_GENERATION', 'use_cython')
File "/home/eogbe/anaconda2/lib/python2.7/ConfigParser.py", line 368, in getboolean
v = self.get(section, option)
File "/home/eogbe/anaconda2/lib/python2.7/ConfigParser.py", line 607, in get
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'CODE_GENERATION'

It seems the ConfigParser cannot find a section. I have no idea how to fix this. I have the HSL MC29 installed on a linux system, running python 2.7.

[dpo]

Did you copy site.template.cfg to site.cfg and edit it to reflect your setup?

[eogbe]

Oh no. I was thinking the default setup should work. I will look into doing that.

[eogbe]

Please where do i find the file site.cfg? I can't see it in the HSL.py directory.

[dpo]

There's a template file named site.template.cfg. Simply make a copy of it and name it site.cfg, then edit it to reflect your local configuration. There are instructions inside.

[eogbe]

@dpo Thanks for the clarification. Now i get the error;

/tmp/tmpbZwxtW/source.c:1:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
void zungqr_();
^
Not building HSL.py, please provide some source files for either mc21, mc29, mc60, ma27 or ma57

I guess the config file cant find the required HSL source code. But i have my downloaded MC29 set at:
hsl_rootdir = ../HSL/MC29. Please what am I not doing right?

Thanks.

[dpo]

@eogbe Could you please paste here your site.cfg?

[eogbe]

# This file provides configuration information about non-Python dependencies for
# HSL.py.

# The format of the file is that of the standard library's ConfigParser module.
#
#   http://www.python.org/doc/current/lib/module-ConfigParser.html
#
# Each section defines settings that apply to one particular dependency. Some of
# the settings are general and apply to nearly any section and are defined here.
# Settings specific to a particular section will be defined near their section.
#
#   libraries
#       Comma-separated list of library names to add to compile the extension
#       with. Note that these should be just the names, not the filenames. For
#       example, the file "libfoo.so" would become simply "foo".
#           libraries = lapack,f77blas,cblas,atlas
#	    libraries = lapack,blas,cblas
#
#   library_dirs
#       List of directories to add to the library search path when compiling
#       extensions with this dependency. Use the character given by os.pathsep
#       to separate the items in the list. On UN*X-type systems (Linux, FreeBSD,
#       OS X):
#           library_dirs = /usr/lib:/usr/local/lib
#       On Windows:
#           library_dirs = c:\mingw\lib,c:\atlas\lib
#
#   include_dirs
#       List of directories to add to the header file earch path.
#           include_dirs = /usr/include:/usr/local/include
#
#   src_dirs
#       List of directories that contain extracted source code for the
#       dependency. For some dependencies, numpy.distutils will be able to build
#       them from source if binaries cannot be found. The FORTRAN BLAS and
#       LAPACK libraries are one example. However, most dependencies are more
#       complicated and require actual installation that you need to do
#       yourself.
#           src_dirs = /home/rkern/src/BLAS_SRC:/home/rkern/src/LAPACK_SRC
#
#   search_static_first
#       Boolean (one of (0, false, no, off) for False or (1, true, yes, on) for
#       True) to tell numpy.distutils to prefer static libraries (.a) over
#       shared libraries (.so). It is turned off by default.
#           search_static_first = false

# Defaults
# ========
# The settings given here will apply to all other sections if not overridden.
# This is a good place to add general library and include directories like
# /usr/local/{lib,include}
#
[DEFAULT]
#library_dirs = /home/MGI/syarre/lib
#include_dirs = /usr/local/include

# Optimized BLAS and LAPACK
# -------------------------
# Use the blas_opt and lapack_opt sections to give any settings that are
# required to link against your chosen BLAS and LAPACK, including the regular
# FORTRAN reference BLAS and also ATLAS. Some other sections still exist for
# linking against certain optimized libraries (e.g. [atlas], [lapack_atlas]),
# however, they are now deprecated and should not be used.
#
# These are typical configurations for ATLAS (assuming that the library and
# include directories have already been set in [DEFAULT]; the include directory
# is important for the BLAS C interface):
#
#[blas_opt]
#libraries = f77blas, cblas, atlas
#
#[lapack_opt]
#libraries = lapack, f77blas, cblas, atlas
#
# If your ATLAS was compiled with pthreads, the names of the libraries might be
# different:
#
#[blas_opt]
#libraries = ptf77blas, ptcblas, atlas
#
#[lapack_opt]
#libraries = lapack, ptf77blas, ptcblas, atlas

# HSL
# ---
# 1. If you wish to build the MA27 interface, you will need the following
#    files from the HSL Archive
#    (see http://www.hsl.rl.ac.uk/archive/hslarchive/hslarchive.html for
#    license terms and for obtaining the software):
#    fd05ad.f  ma27ad.f
#    These two files must be placed in a subdirectory named 'ma27' of the
#    hsl_rootdir directory specified below.
#
# 2. If you wish to build the MA57 interface, you will need the following
#    packages from the HSL:
#    (see http://hsl.rl.ac.uk/hsl2007/hsl20074researchers.html for license
#    terms and for obtaining the software):
#    ma57d mc29d hsl_zd11d.f90
#    The directory specified in hsl_rootdir below should contain subdirectories
#    named ma57 and mc29 containing the contents of the ma57d and mc29d archives.
#    Those should not be altered. The file hsl_zd11d.f90 should be placed in
#    hsl_rootdir.
#
[CODE_GENERATION]
# log file name **without** extension (by default, we use '.log')
log_name = generator
# DEBUG/INFO/WARNING/ERROR/CRITICAL
log_level = DEBUG
console_log_level = DEBUG
file_log_level = DEBUG
# 32bits/64bits
# if left blank, we use INT64_t on 64 bits platforms and INT32_t on 32 bits platforms
DEFAULT_INDEX_TYPE =INT64_t
# Do you want to generate the .c files from Cython?
use_cython = false
# Use debug symbols?
use_debug_symbols = false

# Defaults
# ========
# The settings given here will apply to all other sections if not overridden.
# This is a good place to add general library and include directories like
# /usr/local/{lib,include}
#
[DEFAULT]
library_dirs = /usr/local/lib
include_dirs = /usr/local/include

[HSL]
hsl_rootdir = /home/eogbe/HSL

# METIS
# -----
# You will need the METIS library and header files.
# The source tree may be downloaded from 
# http://glaros.dtc.umn.edu/gkhome/metis/metis/download
# Warning: **only** version **4** is supported by MA57.
[METIS]
metis_dir = /home/eogbe/Documents/myfiles/metis/metis-5.1.0/ 
metis_lib = metis

# Optional
# [CYSPARSE]
# cysparse_rootdir = /Users/syarra/work/VirtualEnvs/nlpy_new/programs/cysparse/

[dpo]

Thank you. Could you please now share the complete output of python setup.py build?

[eogbe]

No problem. Here it is. I didn't change it.

#!/usr/bin/env python

# The file setup.py is automatically generated
# Generate it with
# python generate_code

import pysparse
from distutils.core import setup
from setuptools import find_packages
from distutils.extension import Extension
from numpy.distutils.misc_util import Configuration
from numpy.distutils.system_info import get_info
from numpy.distutils.core import setup


import numpy as np

import ConfigParser
import os
import copy

from codecs import open
from os import path


from subprocess import call

# HELPERS
#--------
def prepare_Cython_extensions_as_C_extensions(extensions):
    """
    Modify the list of sources to transform `Cython` extensions into `C` extensions.
    Args:
        extensions: A list of (`Cython`) `distutils` extensions.
    Warning:
        The extensions are changed in place. This function is not compatible with `C++` code.
    Note:
        Only `Cython` source files are modified into their `C` equivalent source files. Other file types are unchanged.
    """
    for extension in extensions:
        c_sources = list()
        for source_path in extension.sources:
            path, source = os.path.split(source_path)
            filename, ext = os.path.splitext(source)

            if ext == '.pyx':
                c_sources.append(os.path.join(path, filename + '.c'))
            elif ext in ['.pxd', '.pxi']:
                pass
            else:
                # copy source as is
                c_sources.append(source_path)

        # modify extension in place
        extension.sources = c_sources

def files_exist(file_list):
    for fname in file_list:
        if os.path.isfile(fname) == False:
            return False
    return True

f2py_options = []

hsl_config = ConfigParser.SafeConfigParser()
hsl_config.read('site.cfg')

version = {}
with open(os.path.join('hsl', 'version.py')) as fp:
      exec(fp.read(), version)
# later on we use: version['version']

numpy_include = np.get_include()
pysparse_inc = pysparse.get_include()

# Use Cython?
use_cython = hsl_config.getboolean('CODE_GENERATION', 'use_cython')
if use_cython:
    try:
        from Cython.Distutils import build_ext
        from Cython.Build import cythonize
    except ImportError:
        raise ImportError("Check '%s': Cython is not properly installed." % hsl_config)

# DEFAULT
default_include_dir = hsl_config.get('DEFAULT', 'include_dirs').split(os.pathsep)
default_library_dir = hsl_config.get('DEFAULT', 'library_dirs').split(os.pathsep)

# Debug mode
use_debug_symbols = hsl_config.getboolean('CODE_GENERATION', 'use_debug_symbols')

# find user defined directories
hsl_rootdir = hsl_config.get('HSL', 'hsl_rootdir')
if hsl_rootdir == '':
    raise ValueError("Please provide HSL source directory!")

metis_dir = hsl_config.get('METIS', 'metis_dir')
metis_lib = hsl_config.get('METIS', 'metis_lib')

# OPTIONAL
build_cysparse_ext = False
if hsl_config.has_section('CYSPARSE'):
    build_cysparse_ext = True
    cysparse_rootdir = hsl_config.get('CYSPARSE', 'cysparse_rootdir').split(os.pathsep)
    if cysparse_rootdir == '':
        raise ValueError("You must specify where CySparse source code is" +
                         "located. Use `cysparse_rootdir` to specify its path.")

config = Configuration('hsl', '', '')
config.set_options(ignore_setup_xxx_py=True,
                   assume_default_configuration=True,
                   delegate_options_to_subpackages=True,
                   quiet=True)

# Set config.version
config.get_version(os.path.join('hsl', 'version.py'))


config.add_include_dirs([np.get_include(), pysparse.get_include()])
config.add_include_dirs('include')

# Get info from site.cfg
blas_info = get_info('blas_opt', 0)
if not blas_info:
    print 'No blas info found'

lapack_info = get_info('lapack_opt', 0)
if not lapack_info:
    print 'No lapack info found'


########################################################################################################################
# EXTENSIONS
########################################################################################################################
include_dirs = [numpy_include, 'include', '.']

## Extensions using Fortran codes

# Relevant files for building MA27 extension.
ma27_src = ['ma27s.f', 'ma27d.f']
libma27_src = ['ma27fact.f']
pyma27_src = ['ma27_lib.c', 'hsl_alloc.c', '_pyma27.c']

# Build PyMA27
ma27_sources = [os.path.join(hsl_rootdir, 'ma27', 'src', name) for name in ma27_src]
ma27_sources += [os.path.join('hsl', 'solvers', 'src', name) for name in libma27_src]
pyma27_sources = [os.path.join('hsl', 'solvers', 'src', name) for name in pyma27_src]

if files_exist(ma27_sources):

    config.add_library(name='hsl_ma27',
                       sources=ma27_sources,
                       include_dirs=[hsl_rootdir, os.path.join('hsl', 'solvers', 'src')],)

    config.add_extension(name='solvers._pyma27',
                         sources=pyma27_sources,
                         libraries=['hsl_ma27'],
                         include_dirs=['include', os.path.join('hsl','solvers','src')],)

# Build PyMA57
ma57_src = ['ddeps.f', 'ma57d.f', 'sdeps.f', 'ma57s.f']
ma57_sources = [os.path.join(hsl_rootdir, 'ma57', 'src', name) for name in ma57_src]
pyma57_src = ['ma57_lib.c', 'hsl_alloc.c', '_pyma57.c']
pyma57_sources = [os.path.join('hsl','solvers','src',name) for name in pyma57_src]

if files_exist(ma57_sources):
    if not metis_lib:
        ma57_src += ['fakemetis.f']
    config.add_library(name='hsl_ma57',
                       sources=ma57_sources,
                       libraries=[metis_lib],
                       library_dirs=[metis_dir],
                       include_dirs=[hsl_rootdir, os.path.join('hsl', 'solvers', 'src')],)

    config.add_extension(name='solvers._pyma57',
                         sources=pyma57_sources,
                         libraries=[metis_lib,'hsl_ma57'],
                         library_dirs=[metis_dir],
                         include_dirs=['include',os.path.join('hsl','solvers','src')],)

mc29_src = ['mc29d.f', 'mc29s.f']
mc29_sources = [os.path.join(hsl_rootdir, 'mc29', 'src', name) for name in mc29_src]
mc29_sources += [os.path.join('hsl', 'scaling', 'src', 'mc29.pyf')]

if files_exist(mc29_sources):
    config.add_extension(
            name='scaling._mc29',
            sources=mc29_sources,
            include_dirs=[os.path.join('hsl', 'scaling', 'src')],
            extra_link_args=[])

mc21_src = ['mc21d.f', 'mc21s.f']
mc21_sources = [os.path.join(hsl_rootdir, 'mc21', 'src', name) for name in mc21_src]
mc21_sources += [os.path.join('hsl', 'ordering', 'src', 'mc21.pyf')]

if files_exist(mc21_sources):
    config.add_extension(
            name='ordering._mc21',
            sources=mc21_sources,
            include_dirs=[os.path.join('hsl', 'ordering', 'src')],
            extra_link_args=[])


mc60_src = ['mc60d.f', 'mc60s.f']
mc60_sources = [os.path.join(hsl_rootdir, 'mc60', 'src', name) for name in mc60_src]
mc60_sources += [os.path.join('hsl', 'ordering', 'src', 'mc60.pyf')]

if files_exist(mc60_sources):
    config.add_extension(
            name='ordering._mc60',
            sources=mc60_sources,
            include_dirs=[os.path.join('hsl', 'ordering', 'src')],
            extra_link_args=[])

## Extensions using Cython
ext_params = {}
ext_params['include_dirs'] = include_dirs
if not use_debug_symbols:
    ext_params['extra_compile_args'] = ["-O2", '-std=c99', '-Wno-unused-function']
    ext_params['extra_link_args'] = []
else:
    ext_params['extra_compile_args'] = ["-g", '-std=c99', '-Wno-unused-function']
    ext_params['extra_link_args'] = ["-g"]

context_ext_params = copy.deepcopy(ext_params)
if files_exist(ma27_sources):
    cyma27_src_INT32_FLOAT64 = ['ma27_lib.c',
                                              'hsl_alloc.c',
                                              '_cyma27_base_INT32_FLOAT64.c']
    cyma27_sources_INT32_FLOAT64 = [os.path.join('hsl', 'solvers', 'src', name) for name in cyma27_src_INT32_FLOAT64]

    cyma27_base_ext_params_INT32_FLOAT64 = copy.deepcopy(ext_params)
    cyma27_base_ext_params_INT32_FLOAT64['libraries'] = ['hsl_ma27']
    retval = os.getcwd()
    os.chdir('hsl/solvers/src')
    call(['cython', '_cyma27_base_INT32_FLOAT64.pyx'])
    os.chdir(retval)
    config.add_extension(name='solvers.src._cyma27_base_INT32_FLOAT64',
                         sources=cyma27_sources_INT32_FLOAT64,
                         **cyma27_base_ext_params_INT32_FLOAT64)

    retval = os.getcwd()
    os.chdir('hsl/solvers/src')
    call(['cython', '_cyma27_numpy_INT32_FLOAT64.pyx'])
    os.chdir(retval)
    cyma27_numpy_ext_params_INT32_FLOAT64 = copy.deepcopy(ext_params)
    config.add_extension(name="solvers.src._cyma27_numpy_INT32_FLOAT64",
                         sources=['hsl/solvers/src/_cyma27_numpy_INT32_FLOAT64.c'],
                         **cyma27_numpy_ext_params_INT32_FLOAT64)



if files_exist(ma57_sources):
    cyma57_src_INT32_FLOAT64 = ['ma57_lib.c',
                                              'hsl_alloc.c',
                                              '_cyma57_base_INT32_FLOAT64.c']
    cyma57_sources_INT32_FLOAT64 = [os.path.join('hsl', 'solvers', 'src', name) for name in cyma57_src_INT32_FLOAT64]

    base_ext_params_INT32_FLOAT64 = copy.deepcopy(ext_params)
    base_ext_params_INT32_FLOAT64['library_dirs'] = [metis_dir]
    base_ext_params_INT32_FLOAT64['libraries'] = [metis_lib, 'hsl_ma57']
    retval = os.getcwd()
    os.chdir('hsl/solvers/src')
    call(['cython', '_cyma57_base_INT32_FLOAT64.pyx'])
    os.chdir(retval)
    config.add_extension(
                name='solvers.src._cyma57_base_INT32_FLOAT64',
                sources=cyma57_sources_INT32_FLOAT64,
                **base_ext_params_INT32_FLOAT64)

    retval = os.getcwd()
    os.chdir('hsl/solvers/src')
    call(['cython', '_cyma57_numpy_INT32_FLOAT64.pyx'])
    os.chdir(retval)
    numpy_ext_params_INT32_FLOAT64 = copy.deepcopy(ext_params)
    config.add_extension(name="solvers.src._cyma57_numpy_INT32_FLOAT64",
                     sources=['hsl/solvers/src/_cyma57_numpy_INT32_FLOAT64.c'],
                     **numpy_ext_params_INT32_FLOAT64)


if build_cysparse_ext:
    if files_exist(ma27_sources):
        retval = os.getcwd()
        os.chdir('hsl/solvers/src')
        call(['cython', '-I', cysparse_rootdir[0], '_cyma27_cysparse_INT32_FLOAT64.pyx'])
        os.chdir(retval)
        cyma27_cysparse_ext_params_INT32_FLOAT64 = copy.deepcopy(ext_params)
        cyma27_cysparse_ext_params_INT32_FLOAT64['include_dirs'].extend(cysparse_rootdir)
        config.add_extension(name="solvers.src._cyma27_cysparse_INT32_FLOAT64",
                     sources=['hsl/solvers/src/_cyma27_cysparse_INT32_FLOAT64.c'],
                     **cyma27_cysparse_ext_params_INT32_FLOAT64)

    if files_exist(ma57_sources):
        retval = os.getcwd()
        os.chdir('hsl/solvers/src')
        call(['cython', '-I', cysparse_rootdir[0], '_cyma57_cysparse_INT32_FLOAT64.pyx'])
        os.chdir(retval)
        cyma57_cysparse_ext_params_INT32_FLOAT64 = copy.deepcopy(ext_params)
        cyma57_cysparse_ext_params_INT32_FLOAT64['include_dirs'].extend(cysparse_rootdir)
        config.add_extension(name="solvers.src._cyma57_cysparse_INT32_FLOAT64",
                     sources=['hsl/solvers/src/_cyma57_cysparse_INT32_FLOAT64.c'],
                     **cyma57_cysparse_ext_params_INT32_FLOAT64)


packages_list = ['hsl', 'hsl.ordering', 'hsl.scaling', 'hsl.solvers',
                 'hsl.solvers.src', 'hsl.tools']

CLASSIFIERS = """\
Development Status :: 4 - Beta
Intended Audience :: Science/Research
Intended Audience :: Developers
License :: OSI Approved
Programming Language :: Python
Programming Language :: Cython
Topic :: Software Development
Topic :: Scientific/Engineering
Operating System :: POSIX
Operating System :: Unix
Operating System :: MacOS :: MacOS X
Natural Language :: English
"""

here = path.abspath(path.dirname(__file__))
# Get the long description from the relevant file
with open(path.join(here, 'DESCRIPTION.rst'), encoding='utf-8') as f:
    long_description = f.read()

config.make_config_py()

if config.ext_modules == []:
    msg = "Not building HSL.py, please provide some source files for"
    msg += " either mc21, mc29, mc60, ma27 or ma57"
    print msg
else:
    setup(packages=packages_list, zip_safe=False, **config.todict())

[dpo]

Thanks. I know what's in the script, but I would like to see the output.

[eogbe]

Oh this is it.

/tmp/tmpXLcTTu/source.c:1:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
void zungqr_();
^
Not building HSL.py, please provide some source files for either mc21, mc29, mc60, ma27 or ma57

[dpo]

Did you clone the master or develop branch? Everything builds for me on master.

[eogbe]

Yes i cloned from master. I m not sure why it does not work for me. Its not finding the HSL source codes which is strange. I have to figure out why.

[dpo]

This is the important part from site.cfg:

# 1. If you wish to build the MA27 interface, you will need the following
#    files from the HSL Archive
#    (see http://www.hsl.rl.ac.uk/archive/hslarchive/hslarchive.html for
#    license terms and for obtaining the software):
#    fd05ad.f  ma27ad.f
#    These two files must be placed in a subdirectory named 'ma27' of the
#    hsl_rootdir directory specified below.
#
# 2. If you wish to build the MA57 interface, you will need the following
#    packages from the HSL:
#    (see http://hsl.rl.ac.uk/hsl2007/hsl20074researchers.html for license
#    terms and for obtaining the software):
#    ma57d mc29d hsl_zd11d.f90
#    The directory specified in hsl_rootdir below should contain subdirectories
#    named ma57 and mc29 containing the contents of the ma57d and mc29d archives.
#    Those should not be altered. The file hsl_zd11d.f90 should be placed in
#    hsl_rootdir.

I followed that and the build went through. You could try python setup.py build --verbose to try and get more output.

[eogbe]

HSL.py works for MC29 now. I will try it later for the other routines. The problem was from the name of the HSL directory i had. I had a name that didnt match the name mc29 in the config file. Once i changed that, the HSL.py works for the mc29, including the examples. Thanks @dpo for your guidance on the fix.

One last comment, @dpo do you know if the HSL.py can be easily adapted to handle other HSL routines like mc33 not mentioned in HSL.py? I need to use other HSL routines not mentioned in HSL.py.

Thanks as always.