Custom pickler doesn't work in a conda environment (with Python 3.10, at least)

[jpivarski]

Version of Awkward Array

2.4.4 and HEAD

Description and code to reproduce

I've been tracking down my inability to run tests/test_2682_custom_pickler.py locally. It's pretty consistent about working when I don't have dask-awkward installed and not working when I do have dask-awkward installed.

Notes

I haven't seen this issue before, but it's the first time I'm running in Python 3.10 environments. I don't know if that's related. (Or maybe I've never tested it before because I didn't have dask-awkward installed between the time when the custom picklers were added and now.)

Also, I can't say why CI isn't seeing this. Perhaps it has something to do with conda.

Also, I verified that I don't have any self-installed libraries outside of the conda environment. In particular,

importlib.metadata.entry_points(group="awkward.pickle.reduce")

shows nothing outside of my environment and dask-awkward's contribution within it, if dask-awkward has been installed.

Clean-slate test

I created a new conda environment with these initial packages:

  Prefix: /home/jpivarski/mambaforge/envs/just-dask-awkward

  Updating specs:

   - python=3.10
   - dask
   - typing_extensions
   - importlib_metadata
   - numpy
   - packaging


  Package                    Version  Build                Channel           Size
───────────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex                0.1  conda_forge          conda-forge     Cached
  + libstdcxx-ng              13.2.0  h7e041cc_2           conda-forge     Cached
  + python_abi                  3.10  4_cp310              conda-forge     Cached
  + ld_impl_linux-64            2.40  h41732ed_0           conda-forge     Cached
  + ca-certificates        2023.7.22  hbcca054_0           conda-forge     Cached
  + libgomp                   13.2.0  h807b86a_2           conda-forge     Cached
  + _openmp_mutex                4.5  2_gnu                conda-forge     Cached
  + libgcc-ng                 13.2.0  h807b86a_2           conda-forge     Cached
  + rdma-core                   28.9  h59595ed_1           conda-forge     Cached
  + libnuma                   2.0.16  h0b41bf4_1           conda-forge     Cached
  + xorg-libxdmcp              1.1.3  h7f98852_0           conda-forge     Cached
  + pthread-stubs                0.4  h36c2ea0_1001        conda-forge        6kB
  + libev                       4.33  h516909a_1           conda-forge     Cached
  + libgfortran5              13.2.0  ha4646dd_2           conda-forge     Cached
  + gflags                     2.2.2  he1b5a44_1004        conda-forge     Cached
  + libbrotlicommon            1.1.0  hd590300_1           conda-forge     Cached
  + aws-c-common               0.9.3  hd590300_0           conda-forge     Cached
  + keyutils                   1.6.1  h166bdaf_0           conda-forge     Cached
  + xorg-libxau               1.0.11  hd590300_0           conda-forge     Cached
  + lz4-c                      1.9.4  hcb278e6_0           conda-forge     Cached
  + libdeflate                  1.19  hd590300_0           conda-forge       67kB
  + lerc                       4.0.0  h27087fc_0           conda-forge     Cached
  + libjpeg-turbo              3.0.0  hd590300_1           conda-forge      619kB
  + libnsl                     2.0.0  hd590300_1           conda-forge     Cached
  + libffi                     3.4.2  h7f98852_5           conda-forge     Cached
  + bzip2                      1.0.8  h7f98852_4           conda-forge     Cached
  + yaml                       0.2.5  h7f98852_2           conda-forge     Cached
  + libzlib                   1.2.13  hd590300_5           conda-forge     Cached
  + libwebp-base               1.3.2  hd590300_0           conda-forge      402kB
  + libcrc32c                  1.1.2  h9c3ff4c_0           conda-forge     Cached
  + c-ares                    1.19.1  hd590300_0           conda-forge     Cached
  + snappy                    1.1.10  h9fff704_0           conda-forge     Cached
  + re2                   2023.03.02  h8c504da_0           conda-forge     Cached
  + openssl                    3.1.3  hd590300_0           conda-forge     Cached
  + libabseil             20230802.1  cxx17_h59595ed_0     conda-forge     Cached
  + libutf8proc                2.8.0  h166bdaf_0           conda-forge     Cached
  + ncurses                      6.4  hcb278e6_0           conda-forge     Cached
  + libuuid                   2.38.1  h0b41bf4_0           conda-forge     Cached
  + xz                         5.2.6  h166bdaf_0           conda-forge     Cached
  + ucx                       1.14.1  h64cca9d_5           conda-forge     Cached
  + libgfortran-ng            13.2.0  h69a702a_2           conda-forge     Cached
  + glog                       0.6.0  h6f12383_0           conda-forge     Cached
  + libbrotlienc               1.1.0  hd590300_1           conda-forge     Cached
  + libbrotlidec               1.1.0  hd590300_1           conda-forge     Cached
  + aws-c-compression         0.2.17  h184a658_3           conda-forge     Cached
  + aws-checksums             0.1.17  h184a658_2           conda-forge     Cached
  + aws-c-sdkutils            0.1.12  h184a658_2           conda-forge     Cached
  + libxcb                      1.15  h0b41bf4_0           conda-forge     Cached
  + libpng                    1.6.39  h753d276_0           conda-forge     Cached
  + libsqlite                 3.43.0  h2797004_0           conda-forge     Cached
  + tk                        8.6.13  h2797004_0           conda-forge     Cached
  + zstd                       1.5.5  hfc55251_0           conda-forge     Cached
  + libevent                  2.1.12  hf998b51_1           conda-forge     Cached
  + s2n                       1.3.51  h06160fa_0           conda-forge     Cached
  + aws-c-cal                  0.6.2  h09139f6_2           conda-forge     Cached
  + libssh2                   1.11.0  h0841786_0           conda-forge     Cached
  + libnghttp2                1.52.0  h61bc06f_0           conda-forge     Cached
  + libprotobuf               4.23.4  hf27288f_6           conda-forge     Cached
  + libedit             3.1.20191231  he28a2e2_2           conda-forge     Cached
  + readline                     8.2  h8228510_1           conda-forge     Cached
  + libopenblas               0.3.24  pthreads_h413a1c8_0  conda-forge     Cached
  + freetype                  2.12.1  h267a509_2           conda-forge     Cached
  + libtiff                    4.6.0  ha9c0a0a_2           conda-forge      283kB
  + libthrift                 0.19.0  hb90f79a_1           conda-forge     Cached
  + aws-c-io                 0.13.32  h89a0be2_4           conda-forge     Cached
  + orc                        1.9.0  h52d3b3c_2           conda-forge     Cached
  + libgrpc                   1.57.0  ha4d0f93_1           conda-forge     Cached
  + krb5                      1.21.2  h659d440_0           conda-forge     Cached
  + libblas                    3.9.0  18_linux64_openblas  conda-forge     Cached
  + lcms2                       2.15  hb7c19ff_3           conda-forge      240kB
  + openjpeg                   2.5.0  h488ebb8_3           conda-forge      357kB
  + aws-c-event-stream         0.3.2  hd6ebb48_1           conda-forge     Cached
  + aws-c-http                0.7.13  hc690213_1           conda-forge     Cached
  + libcurl                    8.3.0  hca28451_0           conda-forge     Cached
  + libcblas                   3.9.0  18_linux64_openblas  conda-forge     Cached
  + liblapack                  3.9.0  18_linux64_openblas  conda-forge     Cached
  + aws-c-auth                 0.7.4  hc8144f4_1           conda-forge     Cached
  + aws-c-mqtt                 0.9.6  h32970c0_2           conda-forge     Cached
  + libgoogle-cloud           2.12.0  h8d7e28b_2           conda-forge     Cached
  + aws-c-s3                  0.3.17  hb5e3142_3           conda-forge     Cached
  + aws-crt-cpp               0.23.1  h94c364a_5           conda-forge     Cached
  + aws-sdk-cpp             1.11.156  h6600424_3           conda-forge     Cached
  + libarrow                  13.0.0  h1935d02_5_cpu       conda-forge     Cached
  + tzdata                     2023c  h71feb2d_0           conda-forge     Cached
  + python                   3.10.12  hd12c33a_0_cpython   conda-forge     Cached
  + wheel                     0.41.2  pyhd8ed1ab_0         conda-forge     Cached
  + setuptools                68.2.2  pyhd8ed1ab_0         conda-forge     Cached
  + pip                       23.2.1  pyhd8ed1ab_0         conda-forge     Cached
  + six                       1.16.0  pyh6c4a22f_0         conda-forge     Cached
  + pysocks                    1.7.1  pyha2e5f31_6         conda-forge     Cached
  + pytz                2023.3.post1  pyhd8ed1ab_0         conda-forge     Cached
  + python-tzdata             2023.3  pyhd8ed1ab_0         conda-forge     Cached
  + xyzservices             2023.7.0  pyhd8ed1ab_0         conda-forge     Cached
  + fsspec                  2023.9.2  pyh1a96a4e_0         conda-forge     Cached
  + toolz                     0.12.0  pyhd8ed1ab_0         conda-forge     Cached
  + tblib                      2.0.0  pyhd8ed1ab_0         conda-forge     Cached
  + sortedcontainers           2.4.0  pyhd8ed1ab_0         conda-forge     Cached
  + cloudpickle                2.2.1  pyhd8ed1ab_0         conda-forge     Cached
  + click                      8.1.7  unix_pyh707e725_0    conda-forge     Cached
  + zipp                      3.17.0  pyhd8ed1ab_0         conda-forge     Cached
  + packaging                   23.2  pyhd8ed1ab_0         conda-forge     Cached
  + typing_extensions          4.8.0  pyha770c72_0         conda-forge     Cached
  + zict                       3.0.0  pyhd8ed1ab_0         conda-forge     Cached
  + locket                     1.0.0  pyhd8ed1ab_0         conda-forge     Cached
  + python-dateutil            2.8.2  pyhd8ed1ab_0         conda-forge     Cached
  + importlib-metadata         6.8.0  pyha770c72_0         conda-forge     Cached
  + partd                      1.4.1  pyhd8ed1ab_0         conda-forge     Cached
  + importlib_metadata         6.8.0  hd8ed1ab_0           conda-forge        9kB
  + brotli-python              1.1.0  py310hc6cd4ac_1      conda-forge     Cached
  + markupsafe                 2.1.3  py310h2372a71_1      conda-forge     Cached
  + pillow                    10.0.1  py310h01dd4db_2      conda-forge       47MB
  + lz4                        4.3.2  py310h350c4a5_1      conda-forge     Cached
  + tornado                    6.3.3  py310h2372a71_1      conda-forge     Cached
  + pyyaml                     6.0.1  py310h2372a71_1      conda-forge     Cached
  + psutil                     5.9.5  py310h2372a71_1      conda-forge     Cached
  + msgpack-python             1.0.6  py310hd41b1e2_0      conda-forge     Cached
  + numpy                     1.26.0  py310hb13e2d6_0      conda-forge     Cached
  + cytoolz                   0.12.2  py310h2372a71_1      conda-forge     Cached
  + contourpy                  1.1.1  py310hd41b1e2_1      conda-forge     Cached
  + pyarrow                   13.0.0  py310hf9e7431_5_cpu  conda-forge     Cached
  + pandas                     2.1.1  py310hcc13569_1      conda-forge     Cached
  + urllib3                    2.0.6  pyhd8ed1ab_0         conda-forge       98kB
  + jinja2                     3.1.2  pyhd8ed1ab_1         conda-forge     Cached
  + dask-core               2023.9.3  pyhd8ed1ab_0         conda-forge     Cached
  + bokeh                      3.2.2  pyhd8ed1ab_0         conda-forge     Cached
  + distributed             2023.9.3  pyhd8ed1ab_0         conda-forge     Cached
  + dask                    2023.9.3  pyhd8ed1ab_0         conda-forge        7kB

and

  Prefix: /home/jpivarski/mambaforge/envs/just-dask-awkward

  Updating specs:

   - pytest
   - ca-certificates
   - openssl


  Package           Version  Build         Channel           Size
───────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────

  + colorama          0.4.6  pyhd8ed1ab_0  conda-forge     Cached
  + pluggy            1.3.0  pyhd8ed1ab_0  conda-forge     Cached
  + exceptiongroup    1.1.3  pyhd8ed1ab_0  conda-forge     Cached
  + iniconfig         2.0.0  pyhd8ed1ab_0  conda-forge     Cached
  + tomli             2.0.1  pyhd8ed1ab_0  conda-forge     Cached
  + pytest            7.4.2  pyhd8ed1ab_0  conda-forge     Cached

First, I install awkward from conda-forge:

  Install:
────────────────────────────────────────────────────────────────

  + awkward-cpp       24  py310hd41b1e2_0  conda-forge     649kB
  + awkward        2.4.4  pyhd8ed1ab_0     conda-forge     351kB

The tests pass.

Then, I install dask-awkward from conda-forge:

  Install:
────────────────────────────────────────────────────────────────

  + dask-awkward  2023.9.3  pyhd8ed1ab_0  conda-forge     62kB

The tests fail.

Then I remove them both.

  Remove:
─────────────────────────────────────────────────────────────────────

  - dask-awkward  2023.9.3  pyhd8ed1ab_0     conda-forge     Cached
  - awkward          2.4.4  pyhd8ed1ab_0     conda-forge     Cached
  - awkward-cpp         24  py310hd41b1e2_0  conda-forge     Cached

Next, I install awkward from PyPI:

Collecting awkward
  Obtaining dependency information for awkward from https://files.pythonhosted.org/packages/de/5a/7f75daa08ced64a466649133436548d7f830222afa0e2eb6f07c94469a2a/awkward-2.4.4-py3-none-any.whl.metadata
  Using cached awkward-2.4.4-py3-none-any.whl.metadata (6.9 kB)
Collecting awkward-cpp==24 (from awkward)
  Obtaining dependency information for awkward-cpp==24 from https://files.pythonhosted.org/packages/b9/29/63575d5b7cbf6b1e7090078031442975376764ea56670c282ec61edd7ce8/awkward_cpp-24-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached awkward_cpp-24-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.2 kB)
Requirement already satisfied: importlib-metadata>=4.13.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward) (6.8.0)
Requirement already satisfied: numpy>=1.18.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward) (1.26.0)
Requirement already satisfied: packaging in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward) (23.2)
Requirement already satisfied: typing-extensions>=4.1.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward) (4.8.0)
Requirement already satisfied: zipp>=0.5 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from importlib-metadata>=4.13.0->awkward) (3.17.0)
Using cached awkward-2.4.4-py3-none-any.whl (707 kB)
Using cached awkward_cpp-24-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
Installing collected packages: awkward-cpp, awkward
Successfully installed awkward-2.4.4 awkward-cpp-24

The tests pass.

Then I install dask-awkward from PyPI:

Collecting dask-awkward
  Obtaining dependency information for dask-awkward from https://files.pythonhosted.org/packages/7d/4f/c0717deb5e7c167c93f6fcd74a9329ef7560fa829fe2386c6f744eed8270/dask_awkward-2023.9.3-py3-none-any.whl.metadata
  Downloading dask_awkward-2023.9.3-py3-none-any.whl.metadata (3.4 kB)
Requirement already satisfied: awkward>=2.4.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask-awkward) (2.4.4)
Requirement already satisfied: dask>=2023.04.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask-awkward) (2023.9.3)
Requirement already satisfied: typing-extensions>=4.8.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask-awkward) (4.8.0)
Requirement already satisfied: awkward-cpp==24 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward>=2.4.0->dask-awkward) (24)
Requirement already satisfied: importlib-metadata>=4.13.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward>=2.4.0->dask-awkward) (6.8.0)
Requirement already satisfied: numpy>=1.18.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward>=2.4.0->dask-awkward) (1.26.0)
Requirement already satisfied: packaging in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from awkward>=2.4.0->dask-awkward) (23.2)
Requirement already satisfied: click>=8.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask>=2023.04.0->dask-awkward) (8.1.7)
Requirement already satisfied: cloudpickle>=1.5.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask>=2023.04.0->dask-awkward) (2.2.1)
Requirement already satisfied: fsspec>=2021.09.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask>=2023.04.0->dask-awkward) (2023.9.2)
Requirement already satisfied: partd>=1.2.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask>=2023.04.0->dask-awkward) (1.4.1)
Requirement already satisfied: pyyaml>=5.3.1 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask>=2023.04.0->dask-awkward) (6.0.1)
Requirement already satisfied: toolz>=0.10.0 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from dask>=2023.04.0->dask-awkward) (0.12.0)
Requirement already satisfied: zipp>=0.5 in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from importlib-metadata>=4.13.0->awkward>=2.4.0->dask-awkward) (3.17.0)
Requirement already satisfied: locket in /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages (from partd>=1.2.0->dask>=2023.04.0->dask-awkward) (1.0.0)
Downloading dask_awkward-2023.9.3-py3-none-any.whl (72 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.9/72.9 kB 5.8 MB/s eta 0:00:00
Installing collected packages: dask-awkward
Successfully installed dask-awkward-2023.9.3

The tests fail.

Next, I remove them both.

Found existing installation: dask-awkward 2023.9.3
Uninstalling dask-awkward-2023.9.3:
  Would remove:
    /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages/dask_awkward-2023.9.3.dist-info/*
    /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages/dask_awkward/*
Proceed (Y/n)? y
  Successfully uninstalled dask-awkward-2023.9.3
Found existing installation: awkward 2.4.4
Uninstalling awkward-2.4.4:
  Would remove:
    /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages/awkward-2.4.4.dist-info/*
    /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages/awkward/*
Proceed (Y/n)? y
  Successfully uninstalled awkward-2.4.4
Found existing installation: awkward-cpp 24
Uninstalling awkward-cpp-24:
  Would remove:
    /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages/awkward_cpp-24.dist-info/*
    /home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/site-packages/awkward_cpp/*
Proceed (Y/n)? y
  Successfully uninstalled awkward-cpp-24

Now I install awkward from source (main), using the nox, awkward-cpp compilation, install-in-place procedure.

The tests pass.

Then I install dask-awkward from source (main).

The tests fail.

Error messages

When the tests fail, they fail like this:

============================================================= test session starts =============================================================
platform linux -- Python 3.10.12, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/jpivarski/irishep/awkward
configfile: pyproject.toml
collected 4 items                                                                                                                             

tests/test_2682_custom_pickler.py FFFF                                                                                                  [100%]

================================================================== FAILURES ===================================================================
____________________________________________________________ test_default_pickler _____________________________________________________________

    def test_default_pickler():
>       assert _pickle_complex_array_and_return_form_impl() == ak.forms.from_dict(
            {"class": "ListOffsetArray", "offsets": "i64", "content": "int64"}
        )
E       AssertionError: assert ListForm('i64', 'i64', NumpyForm('int64')) == ListOffsetForm('i64', NumpyForm('int64'))
E        +  where ListForm('i64', 'i64', NumpyForm('int64')) = _pickle_complex_array_and_return_form_impl()
E        +  and   ListOffsetForm('i64', NumpyForm('int64')) = <function from_dict at 0x7faed7961510>({'class': 'ListOffsetArray', 'content': 'int64', 'offsets': 'i64'})
E        +    where <function from_dict at 0x7faed7961510> = <module 'awkward.forms' from '/home/jpivarski/irishep/awkward/src/awkward/forms/__init__.py'>.from_dict
E        +      where <module 'awkward.forms' from '/home/jpivarski/irishep/awkward/src/awkward/forms/__init__.py'> = ak.forms


tests/test_2682_custom_pickler.py:44: AssertionError
______________________________________________________________ test_noop_pickler ______________________________________________________________
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/jpivarski/irishep/awkward/tests/test_2682_custom_pickler.py", line 26, in _pickle_complex_array_and_return_form_impl
    return pickle.loads(pickle.dumps(array)).layout.form
  File "/home/jpivarski/irishep/awkward/src/awkward/highlevel.py", line 1468, in __reduce_ex__
    result = custom_reduce(self, protocol)
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 70, in custom_reduce
    plugin = get_custom_reducer()
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 60, in get_custom_reducer
    _plugin = _load_reduce_plugin()
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 43, in _load_reduce_plugin
    raise RuntimeError(
RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint
"""

The above exception was the direct cause of the following exception:

tmp_path = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_noop_pickler0')

    def test_noop_pickler(tmp_path):
>       assert (
            pickle_complex_array_and_return_form(
                """
    def plugin(obj, protocol: int):
        return NotImplemented""",
                tmp_path,
            )
            == ak.forms.from_dict(
                {"class": "ListOffsetArray", "offsets": "i64", "content": "int64"}
            )
        )

tmp_path   = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_noop_pickler0')

tests/test_2682_custom_pickler.py:50: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_2682_custom_pickler.py:40: in pickle_complex_array_and_return_form
    return pickle_future.result()
        executor   = <concurrent.futures.process.ProcessPoolExecutor object at 0x7faed6a29300>
        pickle_future = <Future at 0x7faed6a292d0 state=finished raised RuntimeError>
        pickler_source = '\ndef plugin(obj, protocol: int):\n    return NotImplemented'
        tmp_path   = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_noop_pickler0')
../../mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/_base.py:458: in result
    return self.__get_result()
        self       = None
        timeout    = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = None

    def __get_result(self):
        if self._exception:
            try:
>               raise self._exception
E               RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint

self       = None

../../mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/_base.py:403: RuntimeError
__________________________________________________________ test_non_packing_pickler ___________________________________________________________
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/jpivarski/irishep/awkward/tests/test_2682_custom_pickler.py", line 26, in _pickle_complex_array_and_return_form_impl
    return pickle.loads(pickle.dumps(array)).layout.form
  File "/home/jpivarski/irishep/awkward/src/awkward/highlevel.py", line 1468, in __reduce_ex__
    result = custom_reduce(self, protocol)
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 70, in custom_reduce
    plugin = get_custom_reducer()
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 60, in get_custom_reducer
    _plugin = _load_reduce_plugin()
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 43, in _load_reduce_plugin
    raise RuntimeError(
RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint
"""

The above exception was the direct cause of the following exception:

tmp_path = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_non_packing_pickler0')

    def test_non_packing_pickler(tmp_path):
>       assert (
            pickle_complex_array_and_return_form(
                """
    def plugin(obj, protocol):
        import awkward as ak
        if isinstance(obj, ak.Array):
            form, length, container = ak.to_buffers(obj)
            return (
                object.__new__,
                (ak.Array,),
                (form.to_dict(), length, container, obj.behavior),
            )
        else:
            return NotImplemented""",
                tmp_path,
            )
            == ak.forms.from_dict(
                {"class": "ListArray", "starts": "i64", "stops": "i64", "content": "int64"}
            )
        )

tmp_path   = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_non_packing_pickler0')

tests/test_2682_custom_pickler.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_2682_custom_pickler.py:40: in pickle_complex_array_and_return_form
    return pickle_future.result()
        executor   = <concurrent.futures.process.ProcessPoolExecutor object at 0x7faed6e5c5b0>
        pickle_future = <Future at 0x7faed6e5e710 state=finished raised RuntimeError>
        pickler_source = '\ndef plugin(obj, protocol):\n    import awkward as ak\n    if isinstance(obj, ak.Array):\n        form, length, cont...,\n            (form.to_dict(), length, container, obj.behavior),\n        )\n    else:\n        return NotImplemented'
        tmp_path   = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_non_packing_pickler0')
../../mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/_base.py:458: in result
    return self.__get_result()
        self       = None
        timeout    = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = None

    def __get_result(self):
        if self._exception:
            try:
>               raise self._exception
E               RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint

self       = None

../../mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/_base.py:403: RuntimeError
___________________________________________________________ test_malformed_pickler ____________________________________________________________
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jpivarski/mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/jpivarski/irishep/awkward/tests/test_2682_custom_pickler.py", line 26, in _pickle_complex_array_and_return_form_impl
    return pickle.loads(pickle.dumps(array)).layout.form
  File "/home/jpivarski/irishep/awkward/src/awkward/highlevel.py", line 1468, in __reduce_ex__
    result = custom_reduce(self, protocol)
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 70, in custom_reduce
    plugin = get_custom_reducer()
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 60, in get_custom_reducer
    _plugin = _load_reduce_plugin()
  File "/home/jpivarski/irishep/awkward/src/awkward/_pickle.py", line 43, in _load_reduce_plugin
    raise RuntimeError(
RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint
"""

The above exception was the direct cause of the following exception:

tmp_path = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_malformed_pickler0')

    def test_malformed_pickler(tmp_path):
        with pytest.raises(RuntimeError, match=r"malformed pickler!"):
>           pickle_complex_array_and_return_form(
                """
    def plugin(obj, protocol: int):
        raise RuntimeError('malformed pickler!')""",
                tmp_path,
            )

tmp_path   = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_malformed_pickler0')

tests/test_2682_custom_pickler.py:88: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_2682_custom_pickler.py:40: in pickle_complex_array_and_return_form
    return pickle_future.result()
        executor   = <concurrent.futures.process.ProcessPoolExecutor object at 0x7faed6ea3310>
        pickle_future = <Future at 0x7faed6ea34c0 state=finished raised RuntimeError>
        pickler_source = "\ndef plugin(obj, protocol: int):\n    raise RuntimeError('malformed pickler!')"
        tmp_path   = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_malformed_pickler0')
../../mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/_base.py:458: in result
    return self.__get_result()
        self       = None
        timeout    = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = None

    def __get_result(self):
        if self._exception:
            try:
>               raise self._exception
E               RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint

self       = None

../../mambaforge/envs/just-dask-awkward/lib/python3.10/concurrent/futures/_base.py:403: RuntimeError

During handling of the above exception, another exception occurred:

tmp_path = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_malformed_pickler0')

    def test_malformed_pickler(tmp_path):
>       with pytest.raises(RuntimeError, match=r"malformed pickler!"):
E       AssertionError: Regex pattern did not match.
E        Regex: 'malformed pickler!'
E        Input: 'Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint'

tmp_path   = PosixPath('/tmp/pytest-of-jpivarski/pytest-88/test_malformed_pickler0')

tests/test_2682_custom_pickler.py:87: AssertionError
=========================================================== short test summary info ===========================================================
FAILED tests/test_2682_custom_pickler.py::test_default_pickler - AssertionError: assert ListForm('i64', 'i64', NumpyForm('int64')) == ListOffsetForm('i64', NumpyForm('int64'))
FAILED tests/test_2682_custom_pickler.py::test_noop_pickler - RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint
FAILED tests/test_2682_custom_pickler.py::test_non_packing_pickler - RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint
FAILED tests/test_2682_custom_pickler.py::test_malformed_pickler - AssertionError: Regex pattern did not match.
============================================================== 4 failed in 1.15s ==============================================================

I'm going to see if this can be captured in a GitHub Action.

[jpivarski]

I just tried this with new VMs on AWS, which have no history of ever having Python installed (other than system Python). I tried installing awkward and dask-awkward three ways:

• both as conda-forge packages
• both as PyPI packages in an otherwise empty conda environment (i.e. conda itself was installed, but everything else came in through pip)
• both as PyPI packages on the system Python

In all cases, tests/test_2682_custom_pickler.py passed without dask-awkward installed and failed with dask-awkward installed.

This is very consistent, and now on a platform with no history. I have no idea how our CI manages to not have a problem with this.

[jpivarski]

The reason our CI doesn't catch it is because dask-awkward doesn't get installed in our CI.

[jpivarski]

The issue has been reproduced in CI:

https://github.com/scikit-hep/awkward/actions/runs/6411516875/job/17407129086?pr=2739

=================================== FAILURES ===================================
_____________________________ test_default_pickler _____________________________

    def test_default_pickler():
>       assert _pickle_complex_array_and_return_form_impl() == ak.forms.from_dict(
            {"class": "ListOffsetArray", "offsets": "i64", "content": "int64"}
        )
E       AssertionError: assert ListForm('i64', 'i64', NumpyForm('int64')) == ListOffsetForm('i64', NumpyForm('int64'))
E        +  where ListForm('i64', 'i64', NumpyForm('int64')) = _pickle_complex_array_and_return_form_impl()
E        +  and   ListOffsetForm('i64', NumpyForm('int64')) = <function from_dict at 0x7fdfc7fabec0>({'class': 'ListOffsetArray', 'content': 'int64', 'offsets': 'i64'})
E        +    where <function from_dict at 0x7fdfc7fabec0> = <module 'awkward.forms' from '/home/runner/micromamba/envs/awkward/lib/python3.11/site-packages/awkward/forms/__init__.py'>.from_dict
E        +      where <module 'awkward.forms' from '/home/runner/micromamba/envs/awkward/lib/python3.11/site-packages/awkward/forms/__init__.py'> = ak.forms


tests/test_2682_custom_pickler.py:44: AssertionError
______________________________ test_noop_pickler _______________________________
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/runner/micromamba/envs/awkward/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/awkward/awkward/tests/test_2682_custom_pickler.py", line 26, in _pickle_complex_array_and_return_form_impl
    return pickle.loads(pickle.dumps(array)).layout.form
                        ^^^^^^^^^^^^^^^^^^^

self = None

    def __get_result(self):
        if self._exception:
            try:
>               raise self._exception
E               RuntimeError: Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint

self       = None

../../../micromamba/envs/awkward/lib/python3.11/concurrent/futures/_base.py:401: RuntimeError

During handling of the above exception, another exception occurred:

tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_malformed_pickler0')

    def test_malformed_pickler(tmp_path):
>       with pytest.raises(RuntimeError, match=r"malformed pickler!"):
E       AssertionError: Regex pattern did not match.
E        Regex: 'malformed pickler!'
E        Input: 'Encountered multiple Awkward pickle reducers under the `awkward.pickle.reduce` entrypoint'

tmp_path   = PosixPath('/tmp/pytest-of-runner/pytest-0/test_malformed_pickler0')

tests/test_2682_custom_pickler.py:87: AssertionError
=========================== short test summary info ============================
SKIPPED [1] tests/test_0115_generic_reducer_operation.py:1209: I can't think of a canonical UnionArray (non-mergeable contents) that can be used in a reducer
SKIPPED [1] tests/test_0401_add_categorical_type_for_arrow_dictionary.py:400: Fix issues for categorical type
SKIPPED [1] tests/test_0652_tests_of_complex_numbers.py:218: Remember to implement sorting for complex numbers.
SKIPPED [1] tests/test_1072_sort.py:714: I can't think of a canonical UnionArray (non-mergeable contents) that can be used in sorting
SKIPPED [1] tests/test_1072_sort.py:739: I can't think of a canonical UnionArray (non-mergeable contents) that can be used in sorting
SKIPPED [1] tests/test_1300_awkward_to_cpp_converter_with_cling.py:345: ROOT was compiled without C++17 support
SKIPPED [1] tests/test_1300_awkward_to_cpp_converter_with_cling.py:408: ROOT was compiled without C++17 support
SKIPPED [1] tests/test_1300_awkward_to_cpp_converter_with_cling.py:467: ROOT was compiled without C++17 support
SKIPPED [1] tests/test_1300_awkward_to_cpp_converter_with_cling.py:788: ROOT was compiled without C++17 support
SKIPPED [4] tests/test_1440_start_v2_to_parquet.py:223: Categorical arrays can't roundtrip through Parquet due to ARROW-14525
SKIPPED [2] tests/test_1613_generator_tolayout_records.py:145: ROOT was compiled without C++17 support
SKIPPED [2] tests/test_1613_generator_tolayout_records.py:182: ROOT was compiled without C++17 support
SKIPPED [2] tests/test_1613_generator_tolayout_records.py:202: ROOT was compiled without C++17 support
SKIPPED [1] tests/test_1613_generator_tolayout_records.py:291: the test needs an external data file: see the comments
SKIPPED [1] tests/test_1672_broadcast_parameters.py:9: string broadcasting is broken
SKIPPED [1] tests/test_1672_broadcast_parameters.py:23: string broadcasting is broken
SKIPPED [1] tests/test_1672_broadcast_parameters.py:37: string broadcasting is broken
SKIPPED [1] tests/test_1672_broadcast_parameters.py:55: string broadcasting is broken
SKIPPED [1] tests/test_1764_jax_jacobian.py:14: Jacobian support not implemented
SKIPPED [1] tests/test_2306_cppyy_jit.py:48: Awkward Array can only work with cppyy 3.0.1 or later.
SKIPPED [1] tests/test_2306_cppyy_jit.py:83: Awkward Array can only work with cppyy 3.0.1 or later.
SKIPPED [1] tests/test_2327_array_interface.py:10: could not import 'cupy': No module named 'cupy'
SKIPPED [1] tests/test_2649_dlpack_support.py:18: could not import 'cupy': No module named 'cupy'
============ 4 failed, 2490 passed, 29 skipped in 239.86s (0:03:59) ============

[agoose77]

Ah, this is a testing problem rather than a bug per-se. I think what's happening is that dask-awkward registers its own pickler, in addition to the pickler that is registered by the test itself. As such, we find two picklers, and fail. I think the solution should involve hiding the dask-awkward pickler from our test suite, if we can.