Make it a standalone compiler (for SCons etc.)

[techtonik]

Current binaries available from https://anaconda.org/carlkl/mingwpy/files are packed into wheels, which means they require pip to be installed. If compiler can be distributed as a plain GCC .zip then people using SCons and other similar tools to build Python extensions can use standard ways to specify path to the compiler.

This is useful in situations when Python modules are built only as a part of larger C project (RHVoice, libtorrent, ...) so they can be compiled both by GCC and MSVC. It also can be useful for game engines that use Python as a scripting language and that would like to keep their users safe from complicated details of Python packaging toolchain - python -> distutils -> setuptools -> pip -> wheels.

[matthew-brett]

I think you're proposing also releasing the compilers as zip files, that unpack to something like <unpack-root>\mingwpy\bin etc?

[techtonik]

@matthew-brett yes, that's pretty much how I build Wesnoth on Windows - https://bitbucket.org/techtonik/locally/src/b97e8c997bdcc66c8b1d8c93a901e39b0d37e95e/05.wesnoth.py?fileviewer=file-view-default

[ogrisel]

You can always create a "mingwpy" subfolder and unzip the .whl file there no?

[techtonik]

@ogrisel I am not sure that the project doesn't ship some Python extensions like CFFI does that need to be installed to work.

[carlkl]

@techtonik, mingwpy is basically the customized mingw-w64 toolchain packed into <pythonroot>\share\mingwpy as well as some convinience wrapper executables packed into the Scripts folder responsible to start gcc, ld, ... located in share\mingwpy\bin. It does not contain any python extensions or runtime DLLs.

It is possible to deploy the toolchain as it is as zip-file if needed.

The nice thing with wheels is, that they can be uploaded at Pypi or an anaconda.org. The mingwpy toolchain itself is fully portable and can be extracted even at foldernames containing white spaces. It is not necessary to put its bin folder into the PATH environment as long as your scripts is able to find the executables.

[matthew-brett]

Carl - I think the idea is that it might be useful to remove even the slight inconvenience of renaming .whl to .zip / unpacking / moving the folders around by shipping the same content in a convenient zip file.

[ogrisel]

+1 for releasing a ".zip" or "7z" archive with a simpler folder structure (e.g. a single 'mingwpy' top level folder in the zip archive) but only on the GitHub "releases" page and not on PyPI as the zip extension is expected to distribute source archive with a setup.py on PyPI.

On PyPI we should only release wheel archives.

[carlkl]

Using a standalone compiler is useful for some use cases. But you have to create a importfile for libpython and copy it together with the windows c-runtime import file to the \libs folder as a second step. I guess this step should be added as a postinstall.py into the bin folder, right?

[techtonik]

Because Scripts is added into PATH, is it a good idea to have binaries in this folder to be named gcc? It may break some system builds. How a build tools like SCons can detect that this is a modified version of GCC?

Using a standalone compiler is useful for some use cases. But you have to create a importfile for libpython and copy it together with the windows c-runtime import file to the \libs folder as a second step.

What is libpython and why it needs importfile? Sorry, but I am not familiar with compiling C extensions that much.

[techtonik]

An interesting case is how Git is distributed on Windows. It comes with gcc, ssh and a lot of other unix tools, but the only folder added into path is cmd, which contains only:

git.exe
git-gui.exe
gitk.exe
start-ssh-agent.cmd

[carlkl]

I can't comment on Git for Windows (which one btw?).

gcc.exe in Scripts is a convenience wrapper to start gcc, if the Scripts folder is in the PATH.

Otherwise the mingwpy\bin folder has to be added into the PATH by the developer.

Another complication is, that two import files has to be located in the libs folder. This is possible as these import libraries are copied into this folder during pip install wheel. Using mingwpy together with virtualenv possibly needs more actions,see: #31 (comment)

[techtonik]

Git - official https://git-scm.com/downloads

And about #31 - you now see what I meant by complicated details of Python packaging toolchain. Now it is even more complicated - python -> distutils -> setuptools -> pip -> wheels -> virtualenv. And they way it should be is python -> scons -> wheels.

[carlkl]

Personally I prefer https://git-for-windows.github.io/.

Concerning packaging: there are two constrainst:

1. python distutils and numpy distutils MUST find the appropriate gcc.exe executable without guard. That this happens is the developers responsibility. If you prefer to use a standalone toolchain installed elsewehre YOU are responsible to add the suitable toolchain (bitness, C-runtime support) to the PATH environment.
2. appropiate import libaries MUST be placed into the libs folder BEFORE using distutils or scons. This is because the automatism to create import libraries on the fly is buggy and wrong. So YOU are responsible to create a python import libary and copy it - together with the C-runtime import library from the toolchain - into the correct folder.

Step 1. and 2. can be avoided by installing the mingwpy wheels. A workaround for the problems with virtalenv and mingwpy has to be developed and tested.

There is nothing to argue against the existance of standalone mingwpy toolchains. In such a case the wrapper executables won't be needed in Scripts. But don't forget step 1. and 2.

If you want to propose a usable workflow for install and postinstall such a standalone toolchain, please go.

[techtonik]

@carlkl it is the same Git. =)

1. By adding custom patched gcc into PATH it is potentially breaks non-Python software, so unless there is a plan to integrate changes upstream, I'd like to know how I can detect from a tool like SCons that GCC in PATH is a custom patched version (that is needed in PATH, because distutils is buggy and now updated anymore)?

2. appropiate import libaries MUST be placed into the libs folder BEFORE using distutils or scons. This is because the automatism to create import libraries on the fly is buggy and wrong

Is this automatism weirdness documented somewhere along with the right way?

If you want to propose a usable workflow for install and postinstall such a standalone toolchain, please go.

It would be better to build upon some basic tutorial for building extension rather that starting from the middle. Have you seen anything like that?

[carlkl]

Documenation should be found at https://mingwpy.github.io/ sooner or later. At the moment I focus on some build weirdness and the numpy distutils patch.