Compile Python as a shared library and dynamically load it at runtime

[adewes]

I modified the code such that it build the Python shared library and compiles it to an asm.js module. I added an example that imports this shared library at runtime.

The compilation of the shared library works as does the import, but unfortunately I still get errors when running the code in production:

Summary of changes:

• Added --enable-shared flag to build the shared Python library
• Add an extra build step which invokes emcc to build a JS module from the shared library (using SIDE_MODULE=1 to enable dynamic loading)
• I added an example that loads the dynamic library at runtime

Problems encountered:

• When trying to convert libpython3.5.a into a dynamic ASM library, I get a symbol multiply defined error. Inspecting the .a library with llvm-nm reveals that many symbols are defined more than once. I solved this by explicitly building the .so library (which uses different compiler flags it seems)
• When importing the dynamic library and calling Py_InitializeEx I get the following error: Python error: Py_Initialize: can't import _frozen_importlib. I was not able to find the root cause for this (I checked in the output files in the debug/import_error branch for easier debugging).

Any idea where the problem could be? Would be really great to get DLL loading to work, as this would allow (in my understanding) to dynamically import additional Python modules that rely on C extensions (such as numpy).

[Beuc]

Interesting approach.

Accordingly to https://github.com/kripken/emscripten/wiki/Linking I believe that brings 5-10% slowdown, but that may be worth it.

The Py_Initialize error you get may be caused by not including any Python module (.py/.pyc) whatsoever in your lib_files.

Also one needs to make Python actually compile dlopen support (e.g. add ac_cv_func_dlopen=yes in config.site).

Last it's worth noting that with 1.38.1, which defaults to WASM=1, dynamic linking is broken.

[adewes]

Thanks for that feedback, I'll try your suggestions when I have time!

[Beuc]

I was a bit harsh with my comment about WASM, it has opened issues with dynamic loading but it works.
I could load Cython modules dynamically in my Python 2.7 test.
I updated the Linking wiki page accordingly.

[pmp-p]

hi @Beuc , did you have succes in loading wasm modules >4K since they do not support RTLD_NOW on main thread ?

[Beuc]

I've switched to static modules so as to use the Emterpreter, so I can't test right now.
Also I probably didn't test extensively with Chromium at that time :)
The modules were largely over 4k though (100-1500k).
I'll try next time I have the opportunity.

[Beuc]

Btw I'm currently adding my Python tests & scripts at:
https://www.beuc.net/python-emscripten/python/
https://github.com/python-emscripten/python

[Beuc]

Did some experiments today while cleaning-up my code:

Error in loading dynamic library /lib/python2.7/site-packages/pygame_sdl2/error.so: RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

Looks like that doesn't work with Chromium indeed.
No issues with Firefox.

I asked about it at https://groups.google.com/forum/#!topic/emscripten-discuss/cE3hUV3fDSw

[pmp-p]

exactly what i have hit on chromium 71, is it practical to load the .wasm with a worker while waiting to dlopen() it, seems your position is a "no" ?

i have actually an enormous libpanda3d.js of 20MiB to download/eval after already huge libpython (3.7.2), before i can import anything.

i'm quite sure wasm would be faster but i got lost in contradictory documentation and outdated samples. maybe because of difference beetween browsers ?

[Beuc]

I would suggest posting to the emscripten-discuss list, there are far more people there :)

[pmp-p]

sure but the official response there is "I don't think we can point to a specific time, it depends on a lot of things, in particular on how many people help out and how much - help is always welcome and necessary"

by "sample" i mean things i can work on from to help, as i already have a ffi interface for micropython-wasm ( based on @Brion work there https://github.com/brion/libffi/tree/emscripten-work )

[Beuc]

I mean more people can help and give their opinion, that is a general ML (not just dev).
And this project here seems dead.

[pmp-p]

ok :) and yeah maybe https://pythonclock.org/