JuliaCall with Ubuntu Python-3.10 and import PyCall results in free(): invalid pointer

[jjstickel]

Affects: JuliaCall

Describe the bug

Using JuliaCall with Ubuntu 22.04 system python (python-3.10) results in free(): invalid pointer crash if a Julia package is imported that also imports/uses PyCall. I suspect this is related to the long standing problem where PyJulia cannot use statically linked python (JuliaPy/pyjulia#185). I have confirmed that installing python compiled with dynamically linked libpython resolves the issue. If so, ultimately it is a bug of PyJulia, but I am reporting it here to alert others who might encounter it. Perhaps a warning or documentation is merited.

julia> versioninfo()
Julia Version 1.9.4
Commit 8e5136fa297 (2023-11-14 08:46 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 52 × Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, cascadelake)
  Threads: 1 on 52 virtual cores
Environment:
  JULIA_CONDAPKG_BACKEND = Null
  JULIA_PYTHONCALL_EXE = @PyCall
  JULIA_PKG_USE_CLI_GIT = true

julia> ENV["PYTHON"]
"/usr/bin/python3.10"

(@v1.9) pkg> st
Status `~/.julia/environments/v1.9/Project.toml`
  [438e738f] PyCall v1.96.2
  [6099a3de] PythonCall v0.9.15

$ ldd /usr/bin/python3.10
	linux-vdso.so.1 (0x00007ffe3d1e1000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2fb3f19000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f2fb3ee8000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2fb3ecc000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2fb3c00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2fb4604000)

$ /usr/bin/pip3.10 list | grep julia
julia                     0.6.1
juliacall                 0.9.15
juliapkg                  0.1.10

$ /usr/bin/python3.10
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from juliacall import Main as jl
>>> jl.Pkg.status()
Status `~/.julia/environments/pyjuliapkg/Project.toml`
  [438e738f] PyCall v1.96.2
  [6099a3de] PythonCall v0.9.15
>>> jl.seval("import PyCall")
free(): invalid pointer
Aborted (core dumped)

With dynamically linked libpython:

julia> ENV["PYTHON"]
"/usr/local/bin/python3.11"

$ ldd /usr/local/bin/python3.11
	linux-vdso.so.1 (0x00007ffcb2713000)
	libpython3.11.so.1.0 => /usr/local/lib/libpython3.11.so.1.0 (0x00007f3b1aa00000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3b1a600000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3b1a919000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f3b1b0b5000)

$ /usr/local/bin/pip3.11 list | grep julia
julia            0.6.1
juliacall        0.9.15
juliapkg         0.1.10

$ /usr/local/bin/python3.11
Python 3.11.5 (main, Nov 25 2023, 15:23:23) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from juliacall import Main as jl
>>> jl.Pkg.status()
Status `~/.julia/environments/pyjuliapkg/Project.toml`
  [438e738f] PyCall v1.96.2
  [6099a3de] PythonCall v0.9.15
>>> jl.seval("import PyCall")
>>> jl.seval('PyCall.py"print(1+1)"')
2

[cjdoris]

Just checking - did you rebuild PyCall each time you changed the PYTHON env var?

[jjstickel]

Just checking - did you rebuild PyCall each time you changed the PYTHON env var?

Yes. I repeated going back and forth just now, rebuilding PyCall each time, to confirm.

[cjdoris]

Can you do juliapkg.status() and jl.PythonCall.C.CTX each time to check they are configured how we expect?

Can you also check if PyCall works on its own in Julia using both these Python interpreters?

[jjstickel]

Can you do juliapkg.status() and jl.PythonCall.C.CTX each time to check they are configured how we expect?

$ /usr/local/bin/python3.11
Python 3.11.5 (main, Nov 25 2023, 15:23:23) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from juliacall import Main as jl
>>> jl.Pkg.stat
jl.Pkg.stat(    jl.Pkg.status(
>>> jl.Pkg.status()
Status `~/.julia/environments/pyjuliapkg/Project.toml`
  [438e738f] PyCall v1.96.2
  [6099a3de] PythonCall v0.9.15
>>> jl.PythonCall.C.CTX
Julia:
IOContext{IOBuffer}:
  is_embedded = true
  is_initialized = true
  is_preinitialized = false
  lib_ptr = Ptr{Nothing} @0x00007fefde5b52e0
  exe_path = "/usr/local/bin/python3.11"
  lib_path = missing
  dlopen_flags = 0x00000046
  pyprogname = missing
  pyprogname_w = missing
  pyhome = missing
  pyhome_w = missing
  which = :embedded
  version = v"3.11.5"
  matches_pycall = missing

$ /usr/bin/python3.10
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from juliacall import Main as jl
>>> jl.Pkg.status()
Status `~/.julia/environments/pyjuliapkg/Project.toml`
  [438e738f] PyCall v1.96.2
  [6099a3de] PythonCall v0.9.15
>>> jl.PythonCall.C.CTX
Julia:
IOContext{IOBuffer}:
  is_embedded = true
  is_initialized = true
  is_preinitialized = false
  lib_ptr = Ptr{Nothing} @0x00007fb3c422b2e0
  exe_path = "/usr/bin/python3.10"
  lib_path = missing
  dlopen_flags = 0x00000046
  pyprogname = missing
  pyprogname_w = missing
  pyhome = missing
  pyhome_w = missing
  which = :embedded
  version = v"3.10.12"
  matches_pycall = missing

Can you also check if PyCall works on its own in Julia using both these Python interpreters?

Yes, PyCall works from the Julia REPL for both Python interpreters.