Segmentation fault on package import with macOS Python 3.7 interpreter #272

lohedges · 2019-08-29T13:22:34Z

We are having issues importing our Python package, Sire, using the Python 3.7 interpreter from conda-forge, where a segmentation fault is immediately raised. (The issue seems to occur for all Python 3 versions that I've tested.) The segmentation fault does not occur using the Python 3.7 or 3.7m interpreters from python.org, or with a home-baked Python interpreter. See our issue report for further details.

conda list

# packages in environment at /miniconda3/envs/sire:
#
# Name                    Version                   Build  Channel
boost                     1.70.0           py37haf112f3_1    conda-forge
boost-cpp                 1.70.0               h75728bb_2    conda-forge
bzip2                     1.0.8                h01d97ff_0    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py37_1    conda-forge
cftime                    1.0.3.4         py37h1e5eb4f_1001    conda-forge
curl                      7.65.3               h22ea746_0    conda-forge
dbus                      1.13.6               h2f22bb5_0    conda-forge
expat                     2.2.5             h6de7cb9_1003    conda-forge
fftw3f                    3.3.4                         2    omnia
gettext                   0.19.8.1          h46ab8bc_1002    conda-forge
glib                      2.58.3            h9d45998_1002    conda-forge
gsl                       2.5                  ha2d443c_0    conda-forge
hdf4                      4.2.13            hf3c6af0_1002    conda-forge
hdf5                      1.10.5          nompi_h15a436c_1103    conda-forge
icu                       64.2                 h6de7cb9_1    conda-forge
jpeg                      9c                h1de35cc_1001    conda-forge
krb5                      1.16.3            hcfa6398_1001    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libcurl                   7.65.3               h16faf7d_0    conda-forge
libcxx                    8.0.1                         0    conda-forge
libcxxabi                 8.0.1                         0    conda-forge
libedit                   3.1.20170329      hcfe32e1_1001    conda-forge
libffi                    3.2.1             h6de7cb9_1006    conda-forge
libgfortran               4.0.0                         2    conda-forge
libiconv                  1.15              h01d97ff_1005    conda-forge
liblapack                 3.8.0               12_openblas    conda-forge
libnetcdf                 4.6.2             h1a02027_1002    conda-forge
libopenblas               0.3.7                h4bb4525_1    conda-forge
libpng                    1.6.37               h2573ce8_0    conda-forge
libssh2                   1.8.2                hcdc9a53_2    conda-forge
llvm-openmp               8.0.1                h770b8ee_0    conda-forge
ncurses                   6.1               h0a44026_1002    conda-forge
netcdf4                   1.5.1.2          py37h876e625_1    conda-forge
numpy                     1.17.1           py37h6b0580a_0    conda-forge
openmm                    7.3.1           py37_cuda92_rc_2    omnia
openssl                   1.1.1c               h01d97ff_0    conda-forge
pcre                      8.41              h0a44026_1003    conda-forge
pip                       19.2.3                   py37_0    conda-forge
pyqt                      5.9.2            py37h2a560b1_2    conda-forge
python                    3.7.3                h93065d6_1    conda-forge
qt                        5.9.7                h8cf7e54_3    conda-forge
readline                  8.0                  hcfe32e1_0    conda-forge
setuptools                41.2.0                   py37_0    conda-forge
sip                       4.19.8          py37h0a44026_1000    conda-forge
sire                      2019.1.0        py37h0a44026_56    michellab/label/dev
sqlite                    3.29.0               hb7d70f7_1    conda-forge
tbb                       2019.8               h770b8ee_0    conda-forge
tbb-devel                 2019.8               h770b8ee_0    conda-forge
tk                        8.6.9             h2573ce8_1002    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xz                        5.2.4             h1de35cc_1001    conda-forge
zlib                      1.2.11            h01d97ff_1005    conda-forge

conda info

    active environment : sire
    active env location : /miniconda3/envs/sire
            shell level : 2
       user config file : /Users/lester/.condarc
 populated config files : /Users/lester/.condarc
          conda version : 4.5.12
    conda-build version : not installed
         python version : 3.7.1.final.0
       base environment : /miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/osx-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /miniconda3/pkgs
                          /Users/lester/.conda/pkgs
       envs directories : /miniconda3/envs
                          /Users/lester/.conda/envs
               platform : osx-64
             user-agent : conda/4.5.12 requests/2.21.0 CPython/3.7.1 Darwin/18.7.0 OSX/10.14.6
                UID:GID : 505:20
             netrc file : None
           offline mode : False

lldb backtrace

(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) run
Process 748 launched: '/miniconda3/envs/sire/bin/python' (x86_64)
Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 14:38:56) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import Sire
Process 748 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xa9)
    frame #0: 0x00000001001f1a3d python`visit_decref + 13
python`visit_decref:
->  0x1001f1a3d <+13>: testb  $0x40, 0xa9(%rax)
    0x1001f1a44 <+20>: jne    0x1001f1a4f               ; <+31>
    0x1001f1a46 <+22>: xorl   %eax, %eax
    0x1001f1a48 <+24>: addq   $0x8, %rsp
Target 0: (python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xa9)
  * frame #0: 0x00000001001f1a3d python`visit_decref + 13
    frame #1: 0x00000001000a2fb7 python`tupletraverse + 55
    frame #2: 0x00000001001f0803 python`collect + 323
    frame #3: 0x00000001001f20ab python`_PyObject_GC_Alloc + 315
    frame #4: 0x00000001000a3e37 python`PyTuple_New + 263
    frame #5: 0x0000000101e3845a libboost_python37.dylib`boost::python::tuple boost::python::make_tuple<char const*>(char const* const&) + 26
    frame #6: 0x0000000101e38029 libboost_python37.dylib`boost::python::objects::function::function(boost::python::objects::py_function const&, boost::python::detail::keyword const*, unsigned int) + 569
    frame #7: 0x0000000101e3b0aa libboost_python37.dylib`boost::python::objects::function_object(boost::python::objects::py_function const&, std::__1::pair<boost::python::detail::keyword const*, boost::python::detail::keyword const*> const&) + 58
    frame #8: 0x000000010bb907de _Base.so`boost::python::api::object boost::python::detail::make_function_aux<void (*)(_object*, SireBase::Array2D<double> const&), boost::python::default_call_policies, boost::mpl::vector3<void, _object*, SireBase::Array2D<double> const&>, mpl_::int_<0> >(void (*)(_object*, SireBase::Array2D<double> const&), boost::python::default_call_policies const&, boost::mpl::vector3<void, _object*, SireBase::Array2D<double> const&> const&, std::__1::pair<boost::python::detail::keyword const*, boost::python::detail::keyword const*> const&, mpl_::int_<0>) + 61
    frame #9: 0x000000010bb906c5 _Base.so`void boost::python::init_base<boost::python::init<SireBase::Array2D<double> const&, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >::visit<boost::python::class_<SireBase::Array2D<double>, boost::python::bases<SireBase::Array2DBase, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_>, boost::python::detail::not_specified, boost::python::detail::not_specified> >(boost::python::class_<SireBase::Array2D<double>, boost::python::bases<SireBase::Array2DBase, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_>, boost::python::detail::not_specified, boost::python::detail::not_specified>&) const + 57
    frame #10: 0x000000010bb8dc10 _Base.so`register_Array2D_double__class() + 853
    frame #11: 0x000000010ba9d289 _Base.so`init_module__Base() + 31
    frame #12: 0x0000000101e41d08 libboost_python37.dylib`boost::python::detail::exception_handler::operator()(boost::function0<void> const&) const + 104
    frame #13: 0x000000010ba915d7 _Error.so`bool boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(SireError::invalid_index const&)> >::operator()<bool, boost::python::detail::translate_exception<SireError::invalid_index, void (*)(SireError::invalid_index const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> >(boost::_bi::type<bool>, boost::python::detail::translate_exception<SireError::invalid_index, void (*)(SireError::invalid_index const&)>&, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&>&, long) + 25
    frame #14: 0x000000010ba915b7 _Error.so`boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<SireError::invalid_index, void (*)(SireError::invalid_index const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(SireError::invalid_index const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke(boost::detail::function::function_buffer&, boost::python::detail::exception_handler const&, boost::function0<void> const&) + 39
    frame #15: 0x000000010ba9148d _Error.so`bool boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(SireError::exception const&)> >::operator()<bool, boost::python::detail::translate_exception<SireError::exception, void (*)(SireError::exception const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> >(boost::_bi::type<bool>, boost::python::detail::translate_exception<SireError::exception, void (*)(SireError::exception const&)>&, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&>&, long) + 25
    frame #16: 0x000000010ba9146d _Error.so`boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<SireError::exception, void (*)(SireError::exception const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(SireError::exception const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke(boost::detail::function::function_buffer&, boost::python::detail::exception_handler const&, boost::function0<void> const&) + 39
    frame #17: 0x0000000101e41a7e libboost_python37.dylib`boost::python::handle_exception_impl(boost::function0<void>) + 62
    frame #18: 0x0000000101e42eb9 libboost_python37.dylib`bool boost::python::handle_exception<void (*)()>(void (*)()) + 73
    frame #19: 0x0000000101e42d76 libboost_python37.dylib`boost::python::detail::init_module(PyModuleDef&, void (*)()) + 86
    frame #20: 0x00000001001993dd python`_PyImport_LoadDynamicModuleWithSpec + 557
    frame #21: 0x00000001001987f3 python`_imp_create_dynamic + 243
    frame #22: 0x000000010002dbc3 python`_PyMethodDef_RawFastCallDict + 259
    frame #23: 0x000000010002f2a1 python`PyCFunction_Call + 65
    frame #24: 0x0000000100169778 python`_PyEval_EvalFrameDefault + 46584
    frame #25: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #26: 0x000000010002e983 python`_PyFunction_FastCallKeywords + 195
    frame #27: 0x000000010016b887 python`call_function + 183
    frame #28: 0x00000001001695b9 python`_PyEval_EvalFrameDefault + 46137
    frame #29: 0x000000010002e275 python`function_code_fastcall + 117
    frame #30: 0x000000010016b887 python`call_function + 183
    frame #31: 0x0000000100169523 python`_PyEval_EvalFrameDefault + 45987
    frame #32: 0x000000010002e275 python`function_code_fastcall + 117
    frame #33: 0x000000010016b887 python`call_function + 183
    frame #34: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #35: 0x000000010002e275 python`function_code_fastcall + 117
    frame #36: 0x000000010016b887 python`call_function + 183
    frame #37: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #38: 0x000000010002e275 python`function_code_fastcall + 117
    frame #39: 0x000000010016b887 python`call_function + 183
    frame #40: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #41: 0x000000010002e275 python`function_code_fastcall + 117
    frame #42: 0x0000000100030918 python`object_vacall + 248
    frame #43: 0x0000000100030b23 python`_PyObject_CallMethodIdObjArgs + 243
    frame #44: 0x0000000100191fb3 python`PyImport_ImportModuleLevelObject + 3619
    frame #45: 0x0000000100167425 python`_PyEval_EvalFrameDefault + 37541
    frame #46: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #47: 0x0000000100157d4b python`builtin_exec + 347
    frame #48: 0x000000010002dbc3 python`_PyMethodDef_RawFastCallDict + 259
    frame #49: 0x000000010002f2a1 python`PyCFunction_Call + 65
    frame #50: 0x0000000100169778 python`_PyEval_EvalFrameDefault + 46584
    frame #51: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #52: 0x000000010002e983 python`_PyFunction_FastCallKeywords + 195
    frame #53: 0x000000010016b887 python`call_function + 183
    frame #54: 0x00000001001695b9 python`_PyEval_EvalFrameDefault + 46137
    frame #55: 0x000000010002e275 python`function_code_fastcall + 117
    frame #56: 0x000000010016b887 python`call_function + 183
    frame #57: 0x0000000100169523 python`_PyEval_EvalFrameDefault + 45987
    frame #58: 0x000000010002e275 python`function_code_fastcall + 117
    frame #59: 0x000000010016b887 python`call_function + 183
    frame #60: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #61: 0x000000010002e275 python`function_code_fastcall + 117
    frame #62: 0x000000010016b887 python`call_function + 183
    frame #63: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #64: 0x000000010002e275 python`function_code_fastcall + 117
    frame #65: 0x0000000100030918 python`object_vacall + 248
    frame #66: 0x0000000100030b23 python`_PyObject_CallMethodIdObjArgs + 243
    frame #67: 0x0000000100191fb3 python`PyImport_ImportModuleLevelObject + 3619
    frame #68: 0x0000000100167425 python`_PyEval_EvalFrameDefault + 37541
    frame #69: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #70: 0x0000000100157d4b python`builtin_exec + 347
    frame #71: 0x000000010002dbc3 python`_PyMethodDef_RawFastCallDict + 259
    frame #72: 0x000000010002f2a1 python`PyCFunction_Call + 65
    frame #73: 0x0000000100169778 python`_PyEval_EvalFrameDefault + 46584
    frame #74: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #75: 0x000000010002e983 python`_PyFunction_FastCallKeywords + 195
    frame #76: 0x000000010016b887 python`call_function + 183
    frame #77: 0x00000001001695b9 python`_PyEval_EvalFrameDefault + 46137
    frame #78: 0x000000010002e275 python`function_code_fastcall + 117
    frame #79: 0x000000010016b887 python`call_function + 183
    frame #80: 0x0000000100169523 python`_PyEval_EvalFrameDefault + 45987
    frame #81: 0x000000010002e275 python`function_code_fastcall + 117
    frame #82: 0x000000010016b887 python`call_function + 183
    frame #83: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #84: 0x000000010002e275 python`function_code_fastcall + 117
    frame #85: 0x000000010016b887 python`call_function + 183
    frame #86: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #87: 0x000000010002e275 python`function_code_fastcall + 117
    frame #88: 0x0000000100030918 python`object_vacall + 248
    frame #89: 0x0000000100030b23 python`_PyObject_CallMethodIdObjArgs + 243
    frame #90: 0x0000000100191fb3 python`PyImport_ImportModuleLevelObject + 3619
    frame #91: 0x0000000100156885 python`builtin___import__ + 149
    frame #92: 0x000000010002f314 python`PyCFunction_Call + 180
    frame #93: 0x0000000100169778 python`_PyEval_EvalFrameDefault + 46584
    frame #94: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #95: 0x000000010002e983 python`_PyFunction_FastCallKeywords + 195
    frame #96: 0x000000010016b887 python`call_function + 183
    frame #97: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #98: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #99: 0x000000010002d777 python`_PyFunction_FastCallDict + 231
    frame #100: 0x0000000100030918 python`object_vacall + 248
    frame #101: 0x0000000100030b23 python`_PyObject_CallMethodIdObjArgs + 243
    frame #102: 0x0000000100191533 python`PyImport_ImportModuleLevelObject + 931
    frame #103: 0x0000000100167425 python`_PyEval_EvalFrameDefault + 37541
    frame #104: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #105: 0x0000000100157d4b python`builtin_exec + 347
    frame #106: 0x000000010002dbc3 python`_PyMethodDef_RawFastCallDict + 259
    frame #107: 0x000000010002f2a1 python`PyCFunction_Call + 65
    frame #108: 0x0000000100169778 python`_PyEval_EvalFrameDefault + 46584
    frame #109: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #110: 0x000000010002e983 python`_PyFunction_FastCallKeywords + 195
    frame #111: 0x000000010016b887 python`call_function + 183
    frame #112: 0x00000001001695b9 python`_PyEval_EvalFrameDefault + 46137
    frame #113: 0x000000010002e275 python`function_code_fastcall + 117
    frame #114: 0x000000010016b887 python`call_function + 183
    frame #115: 0x0000000100169523 python`_PyEval_EvalFrameDefault + 45987
    frame #116: 0x000000010002e275 python`function_code_fastcall + 117
    frame #117: 0x000000010016b887 python`call_function + 183
    frame #118: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #119: 0x000000010002e275 python`function_code_fastcall + 117
    frame #120: 0x000000010016b887 python`call_function + 183
    frame #121: 0x00000001001695f4 python`_PyEval_EvalFrameDefault + 46196
    frame #122: 0x000000010002e275 python`function_code_fastcall + 117
    frame #123: 0x0000000100030918 python`object_vacall + 248
    frame #124: 0x0000000100030b23 python`_PyObject_CallMethodIdObjArgs + 243
    frame #125: 0x0000000100191fb3 python`PyImport_ImportModuleLevelObject + 3619
    frame #126: 0x0000000100167425 python`_PyEval_EvalFrameDefault + 37541
    frame #127: 0x000000010015cf5e python`_PyEval_EvalCodeWithName + 414
    frame #128: 0x00000001001c29de python`PyRun_InteractiveOneObjectEx + 1150
    frame #129: 0x00000001001bf631 python`PyRun_InteractiveLoopFlags + 321
    frame #130: 0x00000001001bf480 python`PyRun_AnyFileExFlags + 64
    frame #131: 0x00000001001ede7a python`pymain_main + 11258
    frame #132: 0x000000010000103d python`main + 125
    frame #133: 0x00007fff780c93d5 libdyld.dylib`start + 1

The text was updated successfully, but these errors were encountered:

lohedges · 2019-08-30T08:26:00Z

After further debugging, the segmentation fault is caused by libpython being statically linked into the Python interpreter. I'll peruse this thread to see if I can come up with a workaround for our build.

[closes #292, closes conda-forge/python-feedstock#272]

aeslaughter · 2019-09-12T21:29:10Z

This should be re-opened, we are seeing something very similar for our project. If someone wants to reproduce the error.

# Get the code that is failing
git clone https://github.com/idaholab/moose.git
cd moose
git fetch origin
git checkout -b python-3 f2cf88fa10
cd framework/contrib/hit

# Test Python2.7 (this works fine)
conda create -n py2 python=2 cython
conda activate py2
python --version

git clean -fxd
rm hit2.cpp
make hit.so
python -c "import hit"

# Test Python3.7 (this fails with Segmentation fault)
conda deactivate
conda create -n py3 python=3 cython
conda activate py3
python --version

git clean -fxd
rm hit3.cpp
make hit.so
python -c "import hit"

mingwandroid · 2019-09-12T23:43:38Z

Your reproduction steps are not using our tools and your project uses python*-config --ldflags when linking an extension module which is incorrect. You should only use that when you are linking to an interpreter library (either static or shared). Instead you need to use LDSHARED as mentioned at: https://bugs.python.org/issue18235#msg277944

Use: python -c "import sysconfig; print(sysconfig.get_config_var('LDSHARED'))"

People often get this wrong. There'll be other problems, if you want smooth integration into the conda-forge ecosystem I recommend making a conda package and perhaps submitting it to conda-forge if there isn't one already.

mingwandroid · 2019-09-13T10:57:18Z

# You need to source this for yourself:
export CONDA_BUILD_SYSROOT=/opt/MacOSX10.9.sdk
conda activate base
for pyver in 2 3; do
  [[ -d ${CONDA_PREFIX}/envs/moose.py${pyver} ]] && rm -rf ${CONDA_PREFIX}/envs/moose.py${pyver}
  conda create -y -n moose.py${pyver} python=${pyver} cython clangxx_osx-64 make
done
[[ -d moose ]] || git clone https://github.com/mingwandroid/moose.git
pushd moose
  git checkout -b python-3 origin/python-3
  pushd framework/contrib/hit
    for pyver in 2 3; do
      conda activate moose.py${pyver}
      git clean -fxd
      rm hit${pyver}.cpp
      make hit.so
      python -c "import hit"
    done
  popd
popd

mingwandroid · 2019-09-13T10:59:55Z

From 82ffb5715d983824955f05feb30f80fd3710c497 Mon Sep 17 00:00:00 2001
From: Ray Donnelly <[email protected]>
Date: Fri, 13 Sep 2019 09:26:30 +0200
Subject: [PATCH] Adjust Python extension module linking to use
 sysconfig.get_config_var('LDSHARED')

---
 framework/contrib/hit/Makefile | 5 ++---
 framework/moose.mk             | 2 +-
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/framework/contrib/hit/Makefile b/framework/contrib/hit/Makefile
index b7994264f8..eb8174f187 100644
--- a/framework/contrib/hit/Makefile
+++ b/framework/contrib/hit/Makefile
@@ -13,9 +13,8 @@ endif
 
 $(info Building hit for python $(PYTHON_VERSION) with $(pyconfig))
 
-PYTHONPREFIX ?= `$(pyconfig) --prefix`
 PYTHONCFLAGS ?= `$(pyconfig) --cflags`
-PYTHONLDFLAGS ?= `$(pyconfig) --ldflags`
+PYTHONLDFLAGS ?= `python -c "import sysconfig; print(sysconfig.get_config_var('LDSHARED').replace('-clang', '-clang++'))"`
 HITCPP := hit$(PYTHON_VERSION).cpp
 
 hit: main.cc parse.cc lex.cc braceexpr.cc braceexpr.h lex.h parse.h
@@ -27,7 +26,7 @@ rewrite: rewrite.cc parse.cc lex.cc lex.h parse.h
 bindings: hit.so
 
 hit.so: parse.cc lex.cc braceexpr.cc $(HITCPP)
-	$(CXX) -std=c++11 -w -fPIC -lstdc++ -shared -L$(PYTHONPREFIX)/lib $(PYTHONCFLAGS) $(PYTHONLDFLAGS) $^ -o $@
+	$(PYTHONLDFLAGS) -std=c++11 -w $(PYTHONCFLAGS) $^ -o $@
 
 $(HITCPP): hit.pyx chit.pxd
 	cython -$(PYTHON_VERSION) -o $@ --cplus $<
diff --git a/framework/moose.mk b/framework/moose.mk
index 0056430e2d..4a5b8b17d8 100644
--- a/framework/moose.mk
+++ b/framework/moose.mk
@@ -52,7 +52,7 @@ endif
 
 hit $(pyhit_LIB): $(pyhit_srcfiles)
 	@echo "Building and linking "$@"..."
-	@bash -c '(cd "$(hit_DIR)" && $(libmesh_CXX) -std=c++11 -w -fPIC -lstdc++ -shared -L`$(pyconfig) --prefix`/lib `$(pyconfig) --includes` `$(pyconfig) --ldflags` $^ -o $(pyhit_LIB))'
+	@bash -c '(cd "$(hit_DIR)" && `python -c "import sysconfig; print(sysconfig.get_config_var('LDSHARED').replace('-clang)', '-clang++')"` -std=c++11 -w `$(pyconfig) --cflags` $^ -o $(pyhit_LIB))'
 
 #
 # gtest
-- 
2.23.0

mingwandroid · 2019-09-13T11:02:35Z

In this patch, I would probably change "-clang" to "clang " and "-clang++" to "clang++ " (note the spaces) .. and I'd also use sys.stdout.write() instead of print().

I believe this is most correct way to link a C++ Python extension module now-a-days, and in particular, within our ecosystem it is the only way we can support (since we build a mix of statically and dynamically linked Python interpreters, which is IMHO a good thing as it helps to shake out issues in projects' build systems, such as this one).

mingwandroid · 2019-09-13T11:03:10Z

As to reopening this, I'm against it.

aeslaughter · 2019-09-13T16:05:17Z

@mingwandroid Thanks! This will certainly help us understand the system and I (we) greatly appreciate your help.

mingwandroid · 2019-09-13T16:09:20Z

No problem, it was a pleasure.

mingwandroid · 2019-09-13T16:14:04Z

.. we should try to document this stuff because I've given the same recommendations before, ping @rrigdon.

mingwandroid · 2019-09-14T10:13:48Z

One thing to note is that LDSHARED includes the compiler executable so my change ignores the value of CXX and libmesh_CXX. If you insist on not doing that then you should replace the first word of LDSHARED with an empty string and reinstate the expansion of those two variables just before the new bit. Still I wouldn't recommend this at all. Conda's python operates in two modes (from a building-submodules perspective, switched depending on whether you install our compilers), using system-compilers mode and using conda-compilers mode. I recommend ignoring the former if you can and sticking to the latter. That is the only guaranteed way we can retain the necessary control of the software stack, otherwise if you must use system tools, CentOS6 is the safest distro to use for Intel Linux, and on other architectures, CentOS7 is safest.

bp-kelley · 2020-04-13T19:34:46Z

Hi, I just noticed this when hunting down an issue we've had with conda, as opposed to singling out condo-forge on APPLE, I think you can ask python whether it was built shared or static:

  # determine linkage of python
  execute_process(
      COMMAND
      ${PYTHON_EXECUTABLE} -c "from distutils import sysconfig; print(sysconfig.get_config_var('Py_ENABLE_SHARED'))"
      OUTPUT_VARIABLE Py_ENABLE_SHARED
      OUTPUT_STRIP_TRAILING_WHITESPACE
  )

This might help prevent headaches in the future.

Fixes stanford-centaur/pono#92 It seems that the `target_link_libraries_with_dynamic_lookup` logic doesn't properly discover this requirement on my version of macOS (10.15.6) and conda (conda 4.8.3). This uses the cmake command from conda-forge/python-feedstock#272 to read Py_ENABLE_SHARED. If this config var isn't set, we use `-undefined dynamic_lookup` rather than linking the python library directly

lohedges added a commit to michellab/Sire that referenced this issue Aug 30, 2019

Don't link against libpython when building wrappers on macOS.

d6c8b3b

[closes #292, closes conda-forge/python-feedstock#272]

lohedges closed this as completed Aug 30, 2019

yves-surrel mentioned this issue Feb 5, 2020

ImportError: DLL load failed geggo/gpyfft#45

Open

makaimann mentioned this issue Oct 5, 2020

Python bindings should use -undefined dynamic_lookup when Py_ENABLE_SHARED is 0 stanford-centaur/pono#92

Closed

This was referenced Oct 5, 2020

Use -undefined dynamic_lookup when Py_ENABLE_SHARED is 0 stanford-centaur/smt-switch#106

Closed

Problem with target_link_libraries_with_dynamic_lookup scikit-build/scikit-build#511

Closed

tdwiser mentioned this issue May 24, 2022

Issue with Python tests CRPropa/CRPropa3#316

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault on package import with macOS Python 3.7 interpreter #272

Segmentation fault on package import with macOS Python 3.7 interpreter #272

lohedges commented Aug 29, 2019 •

edited

Loading

lohedges commented Aug 30, 2019

aeslaughter commented Sep 12, 2019

mingwandroid commented Sep 12, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 13, 2019 •

edited

Loading

mingwandroid commented Sep 13, 2019

aeslaughter commented Sep 13, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 14, 2019

bp-kelley commented Apr 13, 2020

Segmentation fault on package import with macOS Python 3.7 interpreter #272

Segmentation fault on package import with macOS Python 3.7 interpreter #272

Comments

lohedges commented Aug 29, 2019 • edited Loading

lohedges commented Aug 30, 2019

aeslaughter commented Sep 12, 2019

mingwandroid commented Sep 12, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 13, 2019 • edited Loading

mingwandroid commented Sep 13, 2019

aeslaughter commented Sep 13, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 13, 2019

mingwandroid commented Sep 14, 2019

bp-kelley commented Apr 13, 2020

lohedges commented Aug 29, 2019 •

edited

Loading

mingwandroid commented Sep 13, 2019 •

edited

Loading