Python dependency version problem

[kmariuszk]

I was trying to use a torch package in a project I'm working on and there's a bug which basically doesn't allow me to use Julia 1.7 with the newest version of a torch, I need to downgrade it to 1.11.0 in order to make it work, however, it causes >=1.8 Julia version to break. Any ideas why a Python dependency may break Julia? Maybe anyone knows anything about conditional dependencies? I couldn't find anything.

This is the error I'm getting (Julia 1.7, the newest version of torch):

signal (6): Aborted
in expression starting at none:0
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f9b8e55e6f5)
unknown function (ip: 0x7f9b8e575d7b)
unknown function (ip: 0x7f9b8e577ac3)
free at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZN4llvm2cl3optINS_15FunctionSummary23ForceSummaryHotnessTypeELb1ENS0_6parserIS3_EEED2Ev at /opt/hostedtoolcache/julia/1.7.3/x64/bin/../lib/julia/libLLVM-12jl.so (unknown line)
__cxa_finalize at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
__do_global_dtors_aux at /opt/hostedtoolcache/julia/1.7.3/x64/bin/../lib/julia/libLLVM-12jl.so (unknown line)
_fini at /opt/hostedtoolcache/julia/1.7.3/x64/bin/../lib/julia/libLLVM-12jl.so (unknown line)
unknown function (ip: 0x7f9b8e51a494)
exit at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:45
unknown function (ip: 0x7f9b8e4fed8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /opt/hostedtoolcache/julia/1.7.3/x64/bin/julia (unknown line)
Allocations: 266882327 (Pool: 266781876; Big: 100451); GC: 117
ERROR: LoadError: Package CounterfactualExplanations errored during testing (received signal: 6)
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /opt/hostedtoolcache/julia/1.7.3/x64/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68
 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
   @ Pkg.Operations /opt/hostedtoolcache/julia/1.7.3/x64/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1672
 [3] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Vector{String}, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::Base.Pairs{Symbol, IOContext{Base.PipeEndpoint}, Tuple{Symbol}, NamedTuple{(:io,), Tuple{IOContext{Base.PipeEndpoint}}}})
   @ Pkg.API /opt/hostedtoolcache/julia/1.7.3/x64/share/julia/stdlib/v1.7/Pkg/src/API.jl:421
 [4] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{Base.PipeEndpoint}, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:coverage, :julia_args, :force_latest_compatible_version), Tuple{Bool, Vector{String}, Bool}}})
   @ Pkg.API /opt/hostedtoolcache/julia/1.7.3/x64/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
 [5] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:coverage, :julia_args, :force_latest_compatible_version), Tuple{Bool, Vector{String}, Bool}}})
   @ Pkg.API /opt/hostedtoolcache/julia/1.7.3/x64/share/julia/stdlib/v1.7/Pkg/src/API.jl:164
 [6] top-level scope
   @ ~/work/_actions/julia-actions/julia-runtest/v1/test_harness.jl:15
 [7] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [8] top-level scope
   @ none:1
in expression starting at /home/runner/work/_actions/julia-actions/julia-runtest/v1/test_harness.jl:7
Error: Process completed with exit code 1.

[laurikskl]

It's a well-known fact that using Python packages in Julia, such as Torch, can cause significant disruptions. The signal abort error you encountered is just one of the many issues that arise due to the fundamental incompatibility between the two languages. The only foolproof solution is to avoid using Python packages altogether and rely solely on Julia's native ecosystem. This will ensure seamless integration and eliminate the need for downgrading or troubleshooting dependency conflicts. Embrace Julia's strength and let go of Python's limitations.

[mestinso]

When do you get this error? On install or when you try to do specific things with pytorch? Do you have a MWE to create the error log? As a quick note/test, I was able to install latest pytorch on julia 1.7 no problems.

[kmariuszk]

I got it when I try to train my neural network with torch.

With the following function being used while testing there's no error:

function train_and_save_model(data::CounterfactualData, model_location::String, pickle_path::String)
    sys = PythonCall.pyimport("sys")

    if !in(model_location, sys.path)
        sys.path.append(model_location)
    end

    importlib = PythonCall.pyimport("importlib")
    neural_network_class = importlib.import_module("neural_network_class")
    importlib.reload(neural_network_class)
    NeuralNetwork = neural_network_class.NeuralNetwork
    model = NeuralNetwork()

    x_python, y_python = CounterfactualExplanations.DataPreprocessing.preprocess_python_data(data)

    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
    loss_fun = torch.nn.BCEWithLogitsLoss()

    # Training
    # for _ in 1:100
    #     # Compute prediction and loss:
    #     output = model(x_python).squeeze()
    #     loss = loss_fun(output, y_python.t())
    #     print(output)
    #     # Backpropagation:
    #     optimizer.zero_grad()
    #     loss.backward()
    #     optimizer.step()
    # end

    torch.save(model, pickle_path)
end

But it breaks once I uncomment any part of the Training loop.

[cjdoris]

Please open an issue, including a full MWE - i.e. all the information needed to reproduce your error (packages and code). Thanks.