Installing Queueing_Tool in PyPy environment

[galenseilis]

I recall from this issue that at one point I had successfully installed queueing_tool for pypy3. But now I seem to be getting the following error:

$ pypy3 -m pip install queueing_tool
Defaulting to user installation because normal site-packages is not writeable
Collecting queueing_tool
  Using cached queueing-tool-1.2.5.tar.gz (1.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
  WARNING: Generating metadata for package queueing_tool produced metadata for project name unknown. Fix your #egg=queueing_tool fragments.
Discarding https://files.pythonhosted.org/packages/6f/4b/e4dfa2a97e76a7447a360cb2e33b3db4ee68fcb737909d74721d7ddfa44d/queueing-tool-1.2.5.tar.gz#sha256=4804f875a9c159bd8a09fb56affc2f295ed31be11988f5da4047c6541f0f55c5 (from https://pypi.org/simple/queueing-tool/) (requires-python:!=3.0,!=3.1,!=3.2,>=2.7): Requested unknown from https://files.pythonhosted.org/packages/6f/4b/e4dfa2a97e76a7447a360cb2e33b3db4ee68fcb737909d74721d7ddfa44d/queueing-tool-1.2.5.tar.gz#sha256=4804f875a9c159bd8a09fb56affc2f295ed31be11988f5da4047c6541f0f55c5 has inconsistent name: filename has 'queueing-tool', but metadata has 'unknown'
  Using cached queueing-tool-1.2.4.tar.gz (269 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: networkx>=1.9 in /usr/lib/python3/dist-packages (from queueing_tool) (2.4)
Requirement already satisfied: numpy>=1.9 in /home/galen/.local/lib/pypy3.8/site-packages (from queueing_tool) (1.24.4)
Building wheels for collected packages: queueing_tool
  Building wheel for queueing_tool (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [61 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/queueing_tool
      copying queueing_tool/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool
      copying queueing_tool/union_find.py -> build/lib.linux-x86_64-3.8/queueing_tool
      creating build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_functions.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_generation.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_wrapper.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_preparation.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      creating build/lib.linux-x86_64-3.8/queueing_tool/network
      copying queueing_tool/network/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool/network
      copying queueing_tool/network/queue_network.py -> build/lib.linux-x86_64-3.8/queueing_tool/network
      creating build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/queue_servers.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/agents.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/queue_extentions.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      running build_ext
      /usr/lib/python3/dist-packages/pythran/tables.py:4520: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
        if not hasattr(numpy, method):
      /usr/lib/python3/dist-packages/pythran/tables.py:4553: FutureWarning: In the future `np.bytes` will be defined as the corresponding NumPy scalar.
        obj = getattr(themodule, elem)
      Compiling queueing_tool/network/priority_queue.pyx because it changed.
      [1/1] Cythonizing queueing_tool/network/priority_queue.pyx
      /home/galen/.local/lib/pypy3.8/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-5k0qnxlt/queueing-tool_72a3cfa4c64c42d99d074a27eac54e0b/queueing_tool/network/priority_queue.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      performance hint: queueing_tool/network/priority_queue.pyx:131:5: Exception check on '_siftdown' will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      performance hint: queueing_tool/network/priority_queue.pyx:156:5: Exception check on '_siftup' will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      performance hint: queueing_tool/network/priority_queue.pyx:189:5: Exception check on 'heapify' will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      performance hint: queueing_tool/network/priority_queue.pyx:40:28: Index should be typed for more efficient access
      performance hint: queueing_tool/network/priority_queue.pyx:70:28: Exception check will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      performance hint: queueing_tool/network/priority_queue.pyx:119:19: Exception check will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      performance hint: queueing_tool/network/priority_queue.pyx:183:13: Exception check will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      performance hint: queueing_tool/network/priority_queue.pyx:192:15: Exception check will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      error: Command "gcc -pthread -DNDEBUG -O2 -fPIC -I/usr/include/pypy3.8 -c queueing_tool/network/priority_queue.c -o build/temp.linux-x86_64-3.8/queueing_tool/network/priority_queue.o" failed with exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for queueing_tool
  Running setup.py clean for queueing_tool
Failed to build queueing_tool
Installing collected packages: queueing_tool
  Running setup.py install for queueing_tool ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for queueing_tool did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      running install
      /usr/lib/python3/dist-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        setuptools.SetuptoolsDeprecationWarning,
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/queueing_tool
      copying queueing_tool/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool
      copying queueing_tool/union_find.py -> build/lib.linux-x86_64-3.8/queueing_tool
      creating build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_functions.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_generation.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_wrapper.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      copying queueing_tool/graph/graph_preparation.py -> build/lib.linux-x86_64-3.8/queueing_tool/graph
      creating build/lib.linux-x86_64-3.8/queueing_tool/network
      copying queueing_tool/network/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool/network
      copying queueing_tool/network/queue_network.py -> build/lib.linux-x86_64-3.8/queueing_tool/network
      creating build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/queue_servers.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/__init__.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/agents.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      copying queueing_tool/queues/queue_extentions.py -> build/lib.linux-x86_64-3.8/queueing_tool/queues
      running build_ext
      /usr/lib/python3/dist-packages/pythran/tables.py:4520: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
        if not hasattr(numpy, method):
      /usr/lib/python3/dist-packages/pythran/tables.py:4553: FutureWarning: In the future `np.bytes` will be defined as the corresponding NumPy scalar.
        obj = getattr(themodule, elem)
      error: Command "gcc -pthread -DNDEBUG -O2 -fPIC -I/usr/include/pypy3.8 -c queueing_tool/network/priority_queue.c -o build/temp.linux-x86_64-3.8/queueing_tool/network/priority_queue.o" failed with exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> queueing_tool

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I realize there is nothing about queueing_tool's features that promises that it should work with PyPy. But it did give some speed improvement so I would like to better understand how to reproduce that state if possible.

[djordon]

Oh interesting, looks like it doesn't like my c extensions (but did at some time in the past?). I have some ideas on how to fix it. When I get some time, I'll take a look. Hopefully I'll be able to report back by Tuesday morning.

As an aside, did you change your gcc compiler between when you tried this before and now? Did your PyPy version change?

[galenseilis]

Oh interesting, looks like it doesn't like my c extensions (but did at some time in the past?). I have some ideas on how to fix it. When I get some time, I'll take a look. Hopefully I'll be able to report back by Tuesday morning.

As an aside, did you change your gcc compiler between when you tried this before and now? Did your PyPy version change?

I would appreciate you looking further into supporting PyPy if you have time. :)

Unfortunately I did not keep notes or a versioned project. It was just a miscellaneous script. But here is the current state in my venv.

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.4.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-XeT9lY/gcc-11-11.4.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-XeT9lY/gcc-11-11.4.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 

$ pypy3 --version
Python 3.8.13 (7.3.9+dfsg-1ubuntu0.1, Nov 15 2022, 06:22:50)
[PyPy 7.3.9 with GCC 11.3.0]

$ pip3 freeze
Ciw==3.1.0
Cython==3.0.6
networkx==3.2.1
numpy==1.26.2
pandas==2.1.4
python-dateutil==2.8.2
pytz==2023.3.post1
queueing-tool==1.2.5
simpy==4.1.1
six==1.16.0
tqdm==4.66.1
tzdata==2023.3

[djordon]

Hey, I clearly missed my deadline, but I haven't forgotten. I'll get to this soon.

[galenseilis]

Hey, I clearly missed my deadline, but I haven't forgotten. I'll get to this soon.

Sounds good. I appreciate you looking into this! :)

[galenseilis]

BTW I ran a few different tools/implementations to simulate a single MM1 queue for 200 time steps (FIFO service discipline). Some of these implementations I found, some I wrote, and others I modified from previously-existing projects.

The following is a histogram (100 bins) of the results of running each implementation and timing the run time. The vertical axis is frequency across repeated runs and the logarithmic horizontal axis shows the amount of time it took the program to run.

Unsurprisingly the C++ (./cpp_MM1) and Rust (./rust_mm1) implementations are faster than the Python implementations. Each of the Simpy (simpy_MM1.py) and Ciw (ciw_MM1.py) implementations were run with the CPython interpreter and the PyPy interpreter. Within each choice of interpreter it was usually the case the Simpy implementation was faster than Ciw.

Queueing_tool (qt_MM1.py) is faster without PyPy enhancement than either SimPy or Ciw are with it. It will be interesting to see how far queueing_tool takes the lead over other Python packages in run time performance if pypy3 is supported.