Unable to opt-out from python from conda-forge channel #369

stemann · 2023-09-23T10:53:50Z

PythonCall enforce that CondaPkg install python and libstdcxx-ng from the conda-forge channel - even if channels and package-specific channels are specified for a package dependent on PythonCall, e.g. with the following Project.toml:

[deps]
PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"

and CondaPkg.toml:

channels = ["anaconda"]

[deps.python]
version = ">=3"
channel = "anaconda"

(also tried channel = "defaults" or channel = "pkgs/main").

CondaPkg is forced to always include the PythonCall channels -c conda-forge, and packages, e.g. python[version='>=3.7,<4',channel='conda-forge',build='*cpython*'] causing package resolution to obviously fail:

 RUN julia --project=/opt/Project --eval '   using Pkg;   Pkg.add("CondaPkg");   using CondaPkg;   CondaPkg.resolve()'
     Updating registry at `/opt/julia_depot/registries/General.toml`
    Resolving package versions...
   No Changes to `/opt/Project/Project.toml`
   No Changes to `/opt/Project/Manifest.toml`
     CondaPkg Found dependencies: /opt/Project/CondaPkg.toml
     CondaPkg Found dependencies: /opt/julia_depot/packages/PythonCall/qTEA1/CondaPkg.toml
     CondaPkg Resolving changes
              + libstdcxx-ng
              + python
     CondaPkg Installing packages
              │ /usr/local/bin/micromamba
              │ install
              │ -y
              │ -p /opt/conda
              │ --override-channels
              │ --no-channel-priority
              │ libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']
              │ python[version='>=3',channel='anaconda']
              │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
              │ -c anaconda
              └ -c conda-forge
 error    libmamba Could not solve for environment specs
     The following packages are incompatible
     ├─ libstdcxx-ng >=3.4,<13.0  is requested and can be installed;
     ├─ python >=3  is installable with the potential options
     │  ├─ python [3.10.0|3.10.3|...|3.9.7], which can be installed;
     │  ├─ python [3.5.4|3.6.3] would require
     │  │  └─ libstdcxx-ng 7.2.0.* , which conflicts with any installable versions previously reported;
     │  └─ python [3.6.7|3.7.1] would require
     │     └─ openssl >=1.1.1,<1.1.2.0a0 , which does not exist (perhaps a missing channel);
     └─ python >=3.7,<4 *cpython* is not installable because it conflicts with any installable versions previously reported.
 critical libmamba Could not solve for environment specs

The text was updated successfully, but these errors were encountered:

stemann · 2023-09-23T10:54:02Z

This is particularly problematic in the example of trying to extend the PyTorch container's conda installation (at /opt/conda), where the gigabyte-sized PyTorch package (with CUDA support) is re-installed (due to the pytorch-container having python from the pkgs/main channel):

 RUN julia --project=/opt/Project --eval '   using Pkg;   Pkg.add("CondaPkg");   using CondaPkg;   CondaPkg.resolve()'
     Updating registry at `/opt/julia_depot/registries/General.toml`
    Resolving package versions...
   No Changes to `/opt/Project/Project.toml`
   No Changes to `/opt/Project/Manifest.toml`
     CondaPkg Found dependencies: /opt/Project/CondaPkg.toml
     CondaPkg Found dependencies: /opt/julia_depot/packages/PythonCall/qTEA1/CondaPkg.toml
     CondaPkg Resolving changes
              + libstdcxx-ng
              + python
              + pytorch
              + pytorch-cuda
     CondaPkg Installing packages
              │ /usr/local/bin/micromamba
              │ install
              │ -y
              │ -p /opt/conda
              │ --override-channels
              │ --no-channel-priority
              │ libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']
              │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
              │ pytorch[version='*',channel='pytorch']
              │ pytorch-cuda[version='11.7',channel='pytorch']
              │ -c conda-forge
              │ -c nvidia
              └ -c pytorch
 
 Transaction
 
   Prefix: /opt/conda
 
   Updating specs:
 
    - conda-forge::libstdcxx-ng[version='>=3.4,<13.0']
    - conda-forge::python[version='>=3.7,<4',build=*cpython*]
    - pytorch::pytorch=*
    - pytorch::pytorch-cuda==11.7
 
 
   Package                Version  Build                         Channel          Size
 ───────────────────────────────────────────────────────────────────────────────────────
   Install:
 ───────────────────────────────────────────────────────────────────────────────────────
 
   + python_abi              3.10  4_cp310                       conda-forge       6kB
   + lzo                     2.10  h516909a_1000                 conda-forge     321kB
   + aws-c-common           0.9.0  hd590300_0                    conda-forge     198kB
   + libexpat               2.5.0  hcb278e6_1                    conda-forge      78kB
   + libzlib               1.2.13  hd590300_5                    conda-forge      62kB
   + keyutils               1.6.1  h166bdaf_0                    conda-forge     118kB
   + libnsl                 2.0.0  h7f98852_0                    conda-forge      31kB
   + aws-c-compression     0.2.17  h4d4d85c_2                    conda-forge      19kB
   + aws-c-sdkutils        0.1.12  h4d4d85c_1                    conda-forge      53kB
   + aws-checksums         0.1.17  h4d4d85c_1                    conda-forge      50kB
   + libsqlite             3.43.0  h2797004_0                    conda-forge     841kB
   + s2n                   1.3.51  h06160fa_0                    conda-forge     375kB
   + aws-c-cal              0.6.2  hc309b26_1                    conda-forge      51kB
   + aws-c-io             0.13.32  h1a03231_3                    conda-forge     154kB
   + aws-c-http            0.7.12  hc865f51_1                    conda-forge     194kB
   + aws-c-event-stream     0.3.2  h2e3709c_0                    conda-forge      54kB
   + aws-c-auth             0.7.3  he2921ad_3                    conda-forge     102kB
   + aws-c-mqtt             0.9.6  h3a0376c_0                    conda-forge     164kB
   + aws-c-s3              0.3.17  h1678ad6_0                    conda-forge      86kB
   + aws-crt-cpp           0.23.1  hf7d0843_2                    conda-forge     324kB
   + aws-sdk-cpp         1.11.156  he6c2984_2                    conda-forge       3MB
 
   Change:
 ───────────────────────────────────────────────────────────────────────────────────────
 
   - _libgcc_mutex            0.1  main                          pkgs/main         3kB
   + _libgcc_mutex            0.1  conda_forge                   conda-forge       3kB
   - zlib                  1.2.13  h5eee18b_0                    pkgs/main       106kB
   + zlib                  1.2.13  hd590300_5                    conda-forge      93kB
   - torchdata              0.6.1  py310                         pytorch           2MB
   + torchdata              0.6.1  py310hbf284fb_1               conda-forge     326kB
 
   Reinstall:
 ───────────────────────────────────────────────────────────────────────────────────────
 
   o pytorch-cuda            11.7  h778d358_5                    pytorch           4kB
   o pytorch                2.0.1  py3.10_cuda11.7_cudnn8.5.0_0  pytorch           1GB
 
   Upgrade:
 ───────────────────────────────────────────────────────────────────────────────────────
 
   - libstdcxx-ng          11.2.0  h1234567_1                    pkgs/main         5MB
   + libstdcxx-ng          12.3.0  h0f45ef3_2                    conda-forge       3MB
   - libgomp               11.2.0  h1234567_1                    pkgs/main       485kB
   + libgomp               13.2.0  h807b86a_2                    conda-forge     421kB
   - libgcc-ng             11.2.0  h1234567_1                    pkgs/main         6MB
   + libgcc-ng             13.2.0  h807b86a_2                    conda-forge     771kB
   - libuv                 1.44.2  h5eee18b_0                    pkgs/main       885kB
   + libuv                 1.46.0  hd590300_0                    conda-forge     893kB
   - icu                     58.2  he6710b0_3                    pkgs/main        11MB
   + icu                     73.2  h59595ed_0                    conda-forge      12MB
   - libiconv                1.16  h7f8727e_2                    pkgs/main       754kB
   + libiconv                1.17  h166bdaf_0                    conda-forge       1MB
   - rhash                  1.4.1  h3c74f83_1                    pkgs/main       208kB
   + rhash                  1.4.4  hd590300_0                    conda-forge     185kB
   - openssl               1.1.1t  h7f8727e_0                    pkgs/main         4MB
   + openssl                3.1.3  hd590300_0                    conda-forge       3MB
   - libuuid               1.41.5  h5eee18b_0                    pkgs/main        28kB
   + libuuid               2.38.1  h0b41bf4_0                    conda-forge      34kB
   - expat                  2.4.9  h6a678d5_0                    pkgs/main       160kB
   + expat                  2.5.0  hcb278e6_1                    conda-forge     137kB
   - libssh2               1.10.0  h8f2d780_0                    pkgs/main       281kB
   + libssh2               1.11.0  h0841786_0                    conda-forge     271kB
   - krb5                  1.19.4  h568e23c_0                    pkgs/main         1MB
   + krb5                  1.21.2  h659d440_0                    conda-forge       1MB
   - libnghttp2            1.46.0  hce63b2e_0                    pkgs/main       696kB
   + libnghttp2            1.52.0  h61bc06f_0                    conda-forge     622kB
   - libxml2               2.10.3  hcbfbd50_0                    pkgs/main       773kB
   + libxml2               2.11.5  h232c23b_1                    conda-forge     706kB
   - python               3.10.11  h7a1cb2a_2                    pkgs/main        28MB
   + python               3.10.12  hd12c33a_0_cpython            conda-forge      26MB
   - libcurl               7.88.1  h91b91d3_0                    pkgs/main       392kB
   + libcurl                8.3.0  hca28451_0                    conda-forge     388kB
   - libarchive             3.6.2  hb4bd9a0_1                    pkgs/main       922kB
   + libarchive             3.7.2  h039dbb9_0                    conda-forge     866kB
   - cmake                 3.22.1  h1fce559_0                    pkgs/main         8MB
   + cmake                 3.27.6  hcfe8598_0                    conda-forge      18MB
 
   Downgrade:
 ───────────────────────────────────────────────────────────────────────────────────────
 
   - _openmp_mutex            5.1  1_gnu                         pkgs/main        21kB
   + _openmp_mutex            4.5  2_gnu                         conda-forge      24kB
 
   Summary:
 
   Install: 21 packages
   Change: 3 packages
   Reinstall: 2 packages
   Upgrade: 18 packages
   Downgrade: 1 packages
 
   Total download: 1GB
 
 ───────────────────────────────────────────────────────────────────────────────────────
 
 
 
 Transaction starting
...

Given the following Project.toml and CondaPkg.toml running in the pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime container:

[deps]
PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"

channels = ["pytorch", "nvidia"]

[deps.pytorch]
channel = "pytorch"

[deps.pytorch-cuda]
channel = "pytorch"
version = "11.7"

Complete Dockerfile for reference:

FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime

RUN DEBIAN_FRONTEND=noninteractive apt-get update \
  && apt-get install -y --no-install-recommends \
    curl \
  && rm -rf /var/lib/apt/lists/*

RUN curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj -C /usr/local bin/micromamba

ENV JULIA_DEPOT_PATH=/opt/julia_depot
RUN curl -fsSL https://install.julialang.org | sh -s -- --default-channel release --path /opt/juliaup --yes && \
  mkdir /opt/Project
ENV PATH=/opt/juliaup/bin:$PATH

COPY Project.toml /opt/Project/
RUN julia --project=/opt/Project --eval ' \
  using Pkg; \
  Pkg.instantiate();'
COPY CondaPkg.toml /opt/Project/
ENV CONDA_PREFIX=/opt/conda
ENV JULIA_CONDAPKG_BACKEND=Current
ENV JULIA_CONDAPKG_ENV=/opt/conda
ENV JULIA_PYTHONCALL_EXE=/opt/conda/bin/python
RUN julia --project=/opt/Project --eval ' \
  using Pkg; \
  Pkg.add("CondaPkg"); \
  using CondaPkg; \
  CondaPkg.resolve()'

cjdoris · 2023-09-29T20:02:58Z

The reason for the seemingly over-specific spec for Python is that conda-forge has both CPython and PyPy available as python and PythonCall requires CPython. To do this requires restricting the build of the python package to *cpython*, which in turn requires the channel to be fixed to conda-forge (because build strings are not consistent between channels).

Unfortunately it does tie you in to the conda-forge channel, which was never the intention.

To allow PythonCall to use Python from another Conda channel, I'd need to treat Python as a special case in CondaPkg - it's certainly doable.

cjdoris · 2023-10-02T16:31:25Z

I've fixed this on the main branch (pkg> add PythonCall#main). You can now install python from conda-forge, anaconda or pkgs/main. It'll be in the next release. Any problems, let me know.

stemann · 2023-10-03T12:31:11Z

I'm not sure this is related, but I just encountered a weird issue, where PythonCall v0.9.14 and CondaPkg v0.2.21 would ensure install of one version of libstdcxx-ng on Julia 1.9.3 (on Ubuntu Linux), and another version of libstdcxx-ng on Julia 1.6.7 (on Ubuntu Linux). MacOS and Windows seems unaffected...

Julia 1.6 libstdcxx-ng dependency: libstdcxx-ng[version='>=3.4,<12.0',channel='conda-forge']
Julia 1.9 libstdcxx-ng dependency: libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']

Causing:

  LoadError: InitError: Python: ImportError: /opt/hostedtoolcache/julia/1.6.7/x64/bin/../lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /tmp/jl_cBW6j8/.CondaPkg/env/lib/python3.10/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-310-x86_64-linux-gnu.so)

in https://github.com/IHPSystems/FiftyOneImageAnnotations.jl/actions/runs/6392741777/job/17350615345?pr=4#step:6:170

CondaPkg.toml:

[deps]
scikit-image = ">=0.1"

[pip.deps]
fiftyone = ">=0.17"

cjdoris · 2023-10-10T07:52:20Z

Please open a new issue.

stemann added the question Further information is requested label Sep 23, 2023

cjdoris closed this as completed Oct 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to opt-out from python from conda-forge channel #369

Unable to opt-out from python from conda-forge channel #369

stemann commented Sep 23, 2023

stemann commented Sep 23, 2023

cjdoris commented Sep 29, 2023

cjdoris commented Oct 2, 2023

stemann commented Oct 3, 2023

cjdoris commented Oct 10, 2023

Unable to opt-out from python from conda-forge channel #369

Unable to opt-out from python from conda-forge channel #369

Comments

stemann commented Sep 23, 2023

stemann commented Sep 23, 2023

cjdoris commented Sep 29, 2023

cjdoris commented Oct 2, 2023

stemann commented Oct 3, 2023

cjdoris commented Oct 10, 2023