Are you switching to dynamic linking?

[tkf]

Issue:

Hi, I'm maintaining a library which requires a Python executable dynamically linked to libpython. It meant that I couldn't support Python installed via conda as it's used to be statically linked. But I just noticed that the latest Python 3.6.6 from conda-forge is dynamically linked. However, 3.7.0 is still statically linked (but it's built before 3.6.6). It would be nice for us if conda-forge distributes dynamically linked Python executables. Could you share your plan? Are you going to switch to dynamic linking? Would it be the case for all platforms?

Just for the record, here is what I get from conda-forge at the moment:

$ conda create --prefix py36 conda-forge::python=3.6
The following NEW packages will be INSTALLED:
...
    python:          3.6.6-h5001a0f_3        conda-forge
...
$ ldd py36/bin/python | grep libpython
        libpython3.6m.so.1.0 => /tmp/py36/bin/../lib/libpython3.6m.so.1.0 (0x00007f65610ef000)
$ conda create --prefix py37 conda-forge::python
...
The following NEW packages will be INSTALLED:
...
    python:          3.7.0-h5001a0f_4        conda-forge
...
$ ldd py37/bin/python | grep libpython  # prints nothing

ref: JuliaPy/pyjulia#185 (comment)

Environment (conda list):

$ conda list
# packages in environment at /home/takafumi/miniconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py36_0
ca-certificates           2018.03.07                    0
certifi                   2018.8.13                py36_0
cffi                      1.11.5           py36h9745a5d_0
chardet                   3.0.4            py36h0f667ec_1
conda                     4.5.10                   py36_0
conda-env                 2.6.0                h36134e3_1
cryptography              2.2.2            py36h14c3975_0
idna                      2.6              py36h82fb2a8_1
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 7.2.0                hdf63c60_3
libstdcxx-ng              7.2.0                hdf63c60_3
ncurses                   6.1                  hf484d3e_0
openssl                   1.0.2p               h14c3975_0
pip                       10.0.1                   py36_0
pycosat                   0.6.3            py36h0a5515d_0
pycparser                 2.18             py36hf9f622e_1
pyopenssl                 18.0.0                   py36_0
pysocks                   1.6.8                    py36_0
python                    3.6.5                hc3d631a_2
readline                  7.0                  ha6073c6_4
requests                  2.18.4           py36he2e5f8d_1
ruamel_yaml               0.15.37          py36h14c3975_2
setuptools                39.2.0                   py36_0
six                       1.11.0           py36h372c433_1
sqlite                    3.23.1               he433501_0
tk                        8.6.7                hc745277_3
urllib3                   1.22             py36hbe7ace6_0
wheel                     0.31.1                   py36_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zlib                      1.2.11               ha838bed_2

Details about conda and system ( conda info ):

$ conda info
     active environment : None
       user config file : /home/takafumi/.condarc
 populated config files :
          conda version : 4.5.10
    conda-build version : not installed
         python version : 3.6.5.final.0
       base environment : /home/takafumi/miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /home/takafumi/miniconda3/pkgs
                          /home/takafumi/.conda/pkgs
       envs directories : /home/takafumi/miniconda3/envs
                          /home/takafumi/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.10 requests/2.18.4 CPython/3.6.5 Linux/4.14.61-1-lts arch/ glibc/2.28
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[bstellato]

I am getting a similar issue on linux where some versions are dynamically linked, e.g., 3.6.6-h5001a0f_3 and if for any reason they get upgraded they are statically linked, e.g., 3.6.7-h0371630_0. Is there a way to know or enforce the installation of a statically vs dynamically linked version?

[scopatz]

libpython.so should be shipped and I think this is what we intend. If it is not present in a package, this is a mistake. A PR fixing the issue would be very helpful! Thanks!

[tkf]

Hi, thanks for the reply. But actually that was not my question. I've never seen libpython missing from the conda package so that's not my complain.

My interest is that if the python executable is dynamically linked to libpython or not. In Linux, you can check it by passing the python executable installed by conda to ldd (please see my examples above).

[jjhelmus]

The python executable is statically linked to libpython in the packages in the gcc7 label and all 3.7.x builds. This is done for improved performance and security. Debian builds Python in a similar manner.

[tkf]

I see. That makes sense. Thanks for the explanation!

[jakirkham]

It should be possible to get all symbols you need from the executable, but you will need to be sure not to use the shared library (which is making me wonder if that shouldn't be dropped).

[jjhelmus]

The shared libpython library is useful when embedding Python in another application. Additional, compiled Python extensions which were built with a pre-static interpreter often link against libpython so the library is needed for compatibility with these extensions.

Debian ships a statically linked python interpreter but libpython can be in installed using apt install libpython-3.5

There was a nice discussion of this topic on the distutils mailing list during the planning for manylinux1 standard.

[tkf]

@jakirkham Yes, we have the code that gets symbols from executable statically linked to libpython. But there is a bit of peculiarity in our setting that prevents us from using it at the moment: https://github.com/JuliaPy/pyjulia/tree/master/julia/fake-julia

@jjhelmus Thanks for the link! It looks like a useful discussion to follow.

[isuruf]

@tkf, here's an idea for your use-case. You can dlopen the executable as they are position independent executables. (conda-forge's python is not, but it soon will be. python in conda-forge/label/gcc7 and anaconda already are position independent executables).

I had to create a symlink python.so -> python to trick Julia's libdl, but it works.

julia> unsafe_string(ccall((:Py_GetVersion, "/home/isuru/miniconda3/envs/py37_anaconda/bin/python"), Ptr{UInt8}, ()))
"3.7.0 (default, Jun 28 2018, 13:15:42) \n[GCC 7.2.0]"

PyCall.jl could load symbols from the python exectuable without launching julia through python. Therefore libpython is not used at all.

[tkf]

@isuruf Awesome! Thanks a lot for looking into this. I didn't know you can do that. It looks like the best solution.

Is there a nice way to detect if a python executable is a position independent executable? Quick googling finds https://unix.stackexchange.com/questions/89211/how-to-test-whether-a-linux-binary-was-compiled-as-position-independent-code but I'm wondering if there is a Python-specific cross-platform way of checking it, like some variable in sysconfig.

cc: @stevengj

[tkf]

detect if a python executable is a position independent executable

I guess just trying to dlopen (and catch the error) would be the simplest?

[isuruf]

I guess just trying to dlopen (and catch the error) would be the simplest?

Yes.

Btw, python2 on Ubuntu 18.04 is a PIE, but python3 is not. (https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1452115/comments/5)

[tkf]

Thanks!

[minrk]

I just ran into this because dolfin's JIT had -lpython unconditionally in its code-generation step, which results in:

Fatal Python error: PyThreadState_Get: no current thread

if libpython is statically linked. Adding the error message here, so folks can find this more easily than I did: libs to be loaded by Python must not link libpython, and instead must rely on -undefined dynamic_lookup (on mac) for symbols in libpython.

When building with Python, it would appear that you can check if it is dynamically linked via sysconfig:

from distutils import sysconfig
sysconfig.get_config_var('Py_ENABLE_SHARED')

[knedlsepp]

Is there any way I can still use LDFLAGS=-Wl,--no-undefined to build CPython extensions with a conda-python interpreter (which was itself built with --disable-shared)?

[mingwandroid]

For any Python interpreter you should be able to use:

python -c "import sysconfig; print(sysconfig.get_config_var('LDSHARED'))"

.. to get the link line for an extension module (it will include the prefixed compiler filename iff you are using our compilers, otherwise it will not). If you are using C++, then replace 'gcc ' with 'g++ ' and 'clang ' with 'clang++ '

[LunarLanding]

What would be the simplest way to get a python executable with enable-shared ? Tried briefly to compile the feedstock locally, but couldn't make it work and was not sure I would get the right output.

I am doing export PY_INTERP_LINKAGE_NATURE=shared; had to edit meta.yaml to disable logic that would set it to '_shared'; however I get errors from cctools-port saying codesign failed.