Help to run PythonCall together with Distributed in a node of a cluster that doesn't have internet conection

[martinmestre]

Hello, I have tracked the source of the error to the following: Julia tries to install CondaPkg. The thing is that I do not use CondaPkg as I have set in a file .conda/envs/.../activate.sh the following:

export JULIA_CONDAPKG_BACKEND="Null"
export JULIA_PYTHONCALL_EXE=$CONDA_PREFIX/bin/python

The error is below. I do not know how to avoid that CondaPkg tries to use internet connection from a node?
Thank you very much.

(cosmos) [mmestre@serafin pipeline_paper]$ cat short_test.out
  Activating project at `~/casa/work/2020/halostream/sources/stream_fit/pipeline_paper`
  Activating project at `~/casa/work/2020/halostream/sources/stream_fit/pipeline_paper`
    CondaPkg Found dependencies: /home/mmestre/.julia/packages/PythonCall/qTEA1/CondaPkg.toml
    CondaPkg Resolving changes
             + libstdcxx-ng
             + python
    CondaPkg Creating environment
             │ /home/mmestre/.julia/artifacts/87052ac9aec71548f804b30280151288cb1ed40e/bin/micromamba
             │ -r /home/mmestre/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root
             │ create
             │ -y
             │ -p /home/mmestre/casa/work/2020/halostream/sources/stream_fit/pipeline_paper/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']
             │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
             └ -c conda-forge
critical libmamba Multiple errors occured:
    Download error (6) Couldn't resolve host name [https://conda.anaconda.org/conda-forge/noarch/repodata.json.zst]
    Could not resolve host: [conda.anaconda.org](http://conda.anaconda.org/)
    Subdir conda-forge/noarch not loaded!
   
error in running finalizer: ErrorException("task switch not allowed from inside gc finalizer")
ijl_error at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/rtutils.c:41
ijl_switch at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:634
try_yieldto at ./task.jl:910
wait at ./task.jl:984
#wait#621 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
wait at ./process.jl:661
jfptr_wait_42367.clone_1 at /home/mmestre/.julia/juliaup/julia-1.9.3+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#7 at /home/mmestre/.julia/packages/SlurmClusterManager/R0zin/src/slurmmanager.jl:90
unknown function (ip: 0x154e6ae440a2)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
run_finalizer at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gc.c:417
jl_gc_run_finalizers_in_list at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gc.c:507
run_finalizers at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gc.c:553
ijl_atexit_hook at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/init.c:299
ijl_exit at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/init.c:207
exit at ./initdefs.jl:28 [inlined]
exec_options at ./client.jl:312
_start at ./client.jl:522
jfptr__start_40034.clone_1 at /home/mmestre/.julia/juliaup/julia-1.9.3+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
true_main at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:573
jl_repl_entrypoint at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:717
main at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/cli/loader_exe.c:59
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
ERROR: LoadError: InitError: failed process: Process(`/home/mmestre/.julia/artifacts/87052ac9aec71548f804b30280151288cb1ed40e/bin/micromamba -r /home/mmestre/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root create -y -p /home/mmestre/casa/work/2020/halostream/sources/stream_fit/pipeline_paper/.CondaPkg/env --override-channels --no-channel-priority "libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']" "python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']" -c conda-forge`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base ./process.jl:477
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:370
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:271
  [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:526
  [7] resolve()
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:384
  [8] envdir()
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/env.jl:70
  [9] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/qTEA1/src/cpython/context.jl:63
 [10] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/qTEA1/src/cpython/CPython.jl:21
 [11] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
 [12] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
 [13] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
 [14] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [16] macro expansion
    @ ./loading.jl:1648 [inlined]
 [17] macro expansion
    @ ./lock.jl:267 [inlined]
 [18] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
 [19] top-level scope

[cjdoris]

What are you running to get that output? You shouldn't be seeing CondaPkg try to install anything if the backend is set to Null - can you check that those env vars are indeed set to the right values right before CondaPkg is loaded?