Handle statically linked case in macOS as well #212

tkf · 2018-10-25T08:11:06Z

It turned out macOS can have statically linked Python when it's installed via conda: #150 (comment)

So it seems linked_libpython (which calls libdl) is the only way to reliably detect if the Python executable is statically linked or not. Since cd2e408 implements it for Windows as well, we can simplify core.py a lot by relying on linked_libpython.

closes #150

It turned out macOS can have statically linked Python when it's installed via conda: JuliaPy#150 (comment) So it seems `linked_libpython` (which calls `libdl`) is the only way to reliably detect if the Python executable is statically linked or not. Since cd2e408 implements it for Windows as well, we can now rely on `linked_libpython` everywhere which simplifies core.py.

coveralls · 2018-10-25T08:27:17Z

Pull Request Test Coverage Report for Build 511

1 of 2 (50.0%) changed or added relevant lines in 1 file are covered.
15 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.1%) to 74.783%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
julia/core.py	1	2	50.0%

Files with Coverage Reduction	New Missed Lines	%
julia/pseudo_python_cli.py	2	72.68%
julia/find_libpython.py	4	73.42%
julia/core.py	9	81.23%

Totals
Change from base Build 509:	0.1%
Covered Lines:	602
Relevant Lines:	805

💛 - Coveralls

coveralls · 2018-10-25T08:27:18Z

Pull Request Test Coverage Report for Build 516

3 of 4 (75.0%) changed or added relevant lines in 1 file are covered.
7 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.3%) to 74.938%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
julia/core.py	3	4	75.0%

Files with Coverage Reduction	New Missed Lines	%
julia/find_libpython.py	2	73.42%
julia/core.py	5	81.64%

Totals
Change from base Build 509:	0.3%
Covered Lines:	601
Relevant Lines:	802

💛 - Coveralls

bstellato · 2018-10-25T14:22:32Z

I have just tried this PR branch but I am getting exactly the same error. It does not change if I delete the .ji file. Maybe adding conda in the CI would help understanding why it happens.

EDIT: I was doing something stupid! It works now 👍 Thanks a lot!

tkf · 2018-10-25T21:30:50Z

@bstellato Thanks for trying it out!

With this PR, I expect that:

python -c "import julia.Main" fails with an informative error message mentioning python-jl.
python-jl -c "import julia.Main" should work fine

where python is installed with conda in macOS.

Do you see this behavior?

bstellato · 2018-10-25T22:39:49Z

This is so weird... it now works for me in both cases! I am using conda, python 3.6 and mac os.
I updated some packages but now linked_libpython() returns

/Users/xxx/miniconda3/envs/python36/lib/libpython3.6m.dylib

I have seen that within my package updates I ended up installing python from conda-forge. If I run conda list | grep python I get

...
python                    3.6.6                h5001a0f_0    conda-forge

Could this be the reason? This solves the problem #185 as well on the linux cluster I am working with.

tkf · 2018-10-26T00:07:20Z

@bstellato It would be nice if you can confirm that python from defaults channel throws a proper exception:

conda create --prefix testpyjulia defaults::python
testpyjulia/bin/pip install 'https://github.com/tkf/pyjulia/archive/macos-conda.zip#egg=julia'
testpyjulia/bin/python -c "import julia.Main"

(or maybe with the old python interpreter you were previously using) which should show something like:

It seems your Julia and PyJulia setup are not supported.

Julia interpreter:
    {runtime}
Python interpreter and libpython used by PyCall.jl:
    {jlinfo.pyprogramname}
    {jl_libpython}
Python interpreter used to import PyJulia and its libpython.
    {sys.executable}
    {py_libpython}

Your Python interpreter "{sys.executable}"
is statically linked to libpython.  Currently, PyJulia does not support
such Python interpreter.  One easy workaround is to run your Python
script with `python-jl` command bundled in PyJulia.  You can simply do:

    python-jl PATH/TO/YOUR/SCRIPT.py

See `python-jl --help` for more information.

For other available workarounds, see:
    https://github.com/JuliaPy/pyjulia/issues/185

Yes, we should be CIing with conda but conda can't be used with tox (until https://github.com/tox-dev/tox-conda is ready) so it's not straightforward to set it up...

bstellato · 2018-10-26T02:00:46Z

I ran the following commands:

conda create -n testpyjulia defaults::python
conda activate testpyjulia
pip install 'https://github.com/tkf/pyjulia/archive/macos-conda.zip#egg=julia'
python -c "import julia.Main"

I get this error though:

(testpyjulia) ➜  ~ python -c "import julia.Main"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/xxx/miniconda3/envs/testpyjulia/lib/python3.7/site-packages/julia/core.py", line 179, in load_module
    JuliaMainModule(self, fullname))
  File "/Users/xxx/miniconda3/envs/testpyjulia/lib/python3.7/site-packages/julia/core.py", line 84, in __init__
    self._julia = loader.julia
  File "/Users/xxx/miniconda3/envs/testpyjulia/lib/python3.7/site-packages/julia/core.py", line 171, in julia
    self.__class__.julia = julia = Julia()
  File "/Users/xxx/miniconda3/envs/testpyjulia/lib/python3.7/site-packages/julia/core.py", line 640, in __init__
    self._call(u"using PyCall")
  File "/Users/xxx/miniconda3/envs/testpyjulia/lib/python3.7/site-packages/julia/core.py", line 696, in _call
    self.check_exception(src)
  File "/Users/xxx/miniconda3/envs/testpyjulia/lib/python3.7/site-packages/julia/core.py", line 746, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'InitError' occurred while calling julia code:
using PyCall

tkf · 2018-10-26T02:40:06Z

Thanks a lot!

But this is not the expected exception... Could you try python -c 'from julia import Julia; Julia(debug=True)'?

Since `if determine_if_statically_linked()` block was removed, we need additional check for statically linked executable. Furthermore, using `find_libpython()` here was not appropriate since `py_libpython` should be the libpython _used_ by the current executable.

tkf · 2018-10-26T06:56:13Z

Actually, it turned out my earlier commit 77577f8 introduced a bug. 2e12d24 should fix it.

Since `normalize_path(None)` is `None`, we don't need the explicit block `if jlinfo.libpython is None`.

No-capturing mode -s is useful for remote debugging especially in case libjulia initialization fails. But it makes reading output almost impossible. Let's use -v instead since it provides what test is being executed in a more readable format.

See: https://travis-ci.org/JuliaPy/pyjulia/jobs/447296193#L1650

tkf added this to the 0.2 milestone Oct 25, 2018

bstellato mentioned this pull request Oct 25, 2018

PyJulia does not work with Julia 1.0 & 0.7 with Python installed via Conda/Ubuntu/(what else?) #185

Open

Remove confusing python-jl --help sentence

bacf5e5

tkf added 3 commits October 26, 2018 00:09

Fix typos and improve docstring

7217d0b

Further simplify is_compatible_exe

f1036a6

Since `normalize_path(None)` is `None`, we don't need the explicit block `if jlinfo.libpython is None`.

Document/comment is_compatible_exe more

65f11f5

tkf force-pushed the macos-conda branch 5 times, most recently from 698e05b to 3c2eee8 Compare October 27, 2018 05:27

Test invoking PyJulia with incompatible Python executable

056a52a

tkf force-pushed the macos-conda branch from 3c2eee8 to 056a52a Compare October 27, 2018 05:42

tkf added 5 commits October 27, 2018 21:30

Refactor is_compatible_exe

2338a82

More documentation in test_compatible_exe.py

32c5d93

Simulate the case PyCall is configured with statically linked Python

44ae919

Use -v instead of -s in CI

5a7cc3c

No-capturing mode -s is useful for remote debugging especially in case libjulia initialization fails. But it makes reading output almost impossible. Let's use -v instead since it provides what test is being executed in a more readable format.

Fix is_dynamically_linked for macOS

a387a3a

See: https://travis-ci.org/JuliaPy/pyjulia/jobs/447296193#L1650

tkf force-pushed the macos-conda branch from 1205834 to a387a3a Compare October 28, 2018 05:25

tkf merged commit f831030 into JuliaPy:master Oct 28, 2018

jakirkham mentioned this pull request Aug 30, 2024

Test failures (cython-related?) on osx-64 with CPython 3.13 in conda-forge numpy/numpy#27314

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle statically linked case in macOS as well #212

Handle statically linked case in macOS as well #212

tkf commented Oct 25, 2018 •

edited

Loading

coveralls commented Oct 25, 2018

coveralls commented Oct 25, 2018 •

edited

Loading

bstellato commented Oct 25, 2018 •

edited

Loading

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 26, 2018

bstellato commented Oct 26, 2018

tkf commented Oct 26, 2018

tkf commented Oct 26, 2018

Handle statically linked case in macOS as well #212

Handle statically linked case in macOS as well #212

Conversation

tkf commented Oct 25, 2018 • edited Loading

coveralls commented Oct 25, 2018

Pull Request Test Coverage Report for Build 511

💛 - Coveralls

coveralls commented Oct 25, 2018 • edited Loading

Pull Request Test Coverage Report for Build 516

💛 - Coveralls

bstellato commented Oct 25, 2018 • edited Loading

tkf commented Oct 25, 2018

bstellato commented Oct 25, 2018

tkf commented Oct 26, 2018

bstellato commented Oct 26, 2018

tkf commented Oct 26, 2018

tkf commented Oct 26, 2018

tkf commented Oct 25, 2018 •

edited

Loading

coveralls commented Oct 25, 2018 •

edited

Loading

bstellato commented Oct 25, 2018 •

edited

Loading