Making dynamic libraries work for Python

[adewes]

First of all: Thanks for building this, it's an amazing project!

I'm trying to use the dynamic linking capabilities of emscripten to compile the Python interpreter as a shared library and import it at runtime. My motivation for this is that I would like to be able to compile Python modules that rely on C extensions (e.g. numpy) and then import these modules dynamically at runtime (which should be possible via dlopen in my understanding).

I started off from from https://github.com/dgym/cpython-emscripten (which works fine when compiling Python statically) and modified the Makefile to build a shared library in addition to the static Python library. This works fine but I run into a problem when importing the library at runtime, as described here:

dgym/cpython-emscripten#1

Before starting to debug this in depth I first wanted to make sure that I correctly set up the build process, hence my question. What I did to generate the dynamic library is the following:

• Compile the Python interpreter into a shared library using emcc (I did not use SIDE_MODULE=1 in the build step for this). This gives me a libpython3.5.so file (which contains LLVM bitcode).
• Converting the shared library into a Javascript library by running emcc -s SIDE_MODULE=1 libpython3.5.so -o libpython3.5.js. This works fine.
• Compiling my example program with MAIN_MODULE=1 without explicitly linking in the static Python library.
• Inserting the following code to import the dynamic library:

EM_ASM({
  Runtime.loadDynamicLibrary('/libpython3.5.js');
}); 

• Initializing the Python interpreter using Py_InitializeEx(0).

Is this the correct way to build and use a dynamic library in this case?

The last step produces an error Python error: Py_Initialize: can't import _frozen_importlib which breaks many things and keeps the interpreter from initializing properly. The core of the interpreter seems to work though (as I can run simple code like 1+1 but not import anything or even print the results).

I created a Gtihub-hosted version with the failing example for easier debugging (it takes a while to load):

http://www.andreas-dewes.de/cpython-emscripten/dll/

The build config for the Python interpreter can be found here:

https://github.com/adewes/cpython-emscripten/tree/master/3.5.2

The build config for the example script is here:

https://github.com/adewes/cpython-emscripten/tree/master/examples/07-dll-loading

Any idea where the problem could be or how I could better debug this? Please let me know if you need any more information, it would really be great to get this working!

In general, is my understanding correct that it should be possible to dynamically load Python libraries through dlopen?

Emscripten version information:

clang version 4.0.0  (emscripten 1.37.13 : 1.37.13)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/[....]/emsdk-portable/clang/e1.37.13_64bit
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6.2.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5.4.1
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.2.0
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.2.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Candidate multilib: x32;@mx32
Selected multilib: .;@m64
INFO:root:(Emscripten: Running sanity checks)

[adewes]

I compiled the library with more debug information now, yielding a slightly better error message (but still nothing that would tell me where the problem lies):

Invalid function pointer called with signature 'iii'. Perhaps this is an invalid value (e.g. caused by calling a virtual method on a NULL pointer)? Or calling a function with an incorrect type, which will fail? (it is worth building your source files with -Werror (warnings are errors), as warnings can indicate undefined behavior which can cause this)
shell_printErr @ python.asm.js:1
nullFunc_iii @ VM818:914
b9 @ VM818:586588
mftCall_iii @ VM818:586509
_getset_get @ VM818:52701
mftCall_iiii @ VM818:586482
__PyObject_GenericGetAttrWithDict @ VM818:125666
_PyObject_GenericGetAttr @ VM818:125829
mftCall_iii @ VM818:586509
_PyObject_GetAttr @ VM818:124986
_builtin_hasattr @ VM818:292395
mftCall_iii @ VM818:586509
_PyCFunction_Call @ VM818:120348
_PyEval_EvalFrameEx @ VM818:302531
_fast_function @ VM818:307523
_PyEval_EvalFrameEx @ VM818:302880
__PyEval_EvalCodeWithName @ VM818:296009
_fast_function @ VM818:307570
_PyEval_EvalFrameEx @ VM818:302880
__PyEval_EvalCodeWithName @ VM818:296009
_PyEval_EvalCodeEx @ VM818:307801
_builtin___build_class__ @ VM818:291212
mftCall_iiii @ VM818:586482
_PyCFunction_Call @ VM818:120329
_PyEval_EvalFrameEx @ VM818:302531
__PyEval_EvalCodeWithName @ VM818:296009
_PyEval_EvalCode @ VM818:295143
_exec_code_in_module @ VM818:344931
_PyImport_ImportFrozenModuleObject @ VM818:345250
_PyImport_ImportFrozenModule @ VM818:345312
_import_init @ VM818:359975
__Py_InitializeEx_Private @ VM818:359695
_Py_InitializeEx @ VM818:361081
asm._Py_InitializeEx @ VM818:590981
_Py_InitializeEx @ python.asm.js:1
OD @ python.asm.js:18
asm._main @ python.asm.js:30
callMain @ python.asm.js:30
doRun @ python.asm.js:30
run @ python.asm.js:30
runCaller @ python.asm.js:30
removeRunDependency @ python.asm.js:1
processPackageData @ python.asm.js:1
(anonymous) @ python.asm.js:1
xhr.onload @ python.asm.js:1
python.asm.js:1 Build with ASSERTIONS=2 for more info.

[adewes]

I now traced the error back to the following CPython code:

https://github.com/python/cpython/blob/fff9a31a91283c39c363af219e595eab7d4da6f7/Objects/descrobject.c#L155

I have no idea why this function is NULL, as Python explicitly checks that it is not NULL before calling it...

[adewes]

I managed to import a Python C extension compiled to a dynamic library from a statically compiled Python now: https://github.com/adewes/cpython-emscripten/tree/master/examples/08-load-dynamic-module

Still struggling to get Python to work as a shared library though, which would be even nicer.

Being able to compile and run C extensions inside the browser would really be amazing, as it would allow us to potentially use a wide range of very efficient Python libraries such as numpy or scipy directly inside the browser, so it would be great if we can make it happen!

[kripken]

repl.it had module loading working in python a while ago - code might still be useful, not sure.

Btw, why build python itself as a side module? It seems simpler to build the python interpreter as a regular program (main module), and build modules to be loaded at runtime as shared libraries (side modules).

That crash says Invalid function pointer called with signature 'iii', so this is likely asm.js/wasm function pointer casting rules. I.e., it's possible python is relying on undefined behavior and calling a function pointer with the wrong type. That happens to work natively, but not on platforms that depend on the type being accurate. See that link for workarounds (like EMULATE_FUNCTION_POINTER_CASTS) and suggestions.

[adewes]

Thanks I'll check out the code from repl.it and will also look at the workarounds.

[takahashi64]

It seems simpler to build the python interpreter as a regular program (main module), and build modules to be loaded at runtime as shared libraries (side modules).

@adewes @kripken I tried this approach, but ran into this issue (I can documented the steps I took if you want):
#5233

[rdb]

Python happens to be a terrible offender of the function pointer casting rules, and requires extensive patching. I happen to be sitting on a patched copy of Python 2.7 (example of it in action) so let me know if you'd like me to share it.

They've made embedding significantly harder in Python 3 because the interpreter needs certain modules to be available and frozen for the initialize function to even work. However it is still doable; you just have to write extra C (or JS) code to initialise PyImport_FrozenModules with appropriate pointers to the bytecode of these modules. The modules to freeze are importlib._bootstrap (as _frozen_importlib) and importlib._bootstrap_external (as _frozen_importlib_external).

In fact, it's probably easier to freeze the entire standard library and any other Python modules you might need, while you're at it (as done in the linked example above). This saves a lot of initialisation time.

[adewes]

Thanks for this, I didn't have time to look further into this last month but I hope to get back to it eventually, this information should be helpful when doing so!

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.