Supporting NumPy 2.0

[qiyunzhu]

See here. To test whether this patch will work, we need to wait until h5py 3.11.0 is released. At present, it will say:

  File "biom/_filter.pyx", line 1, in init biom._filter
    # -----------------------------------------------------------------------------
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

[wasade]

Thanks! I looked at a few of the CI runs, and they looked to be using numpy 1.0. Is there a standard practice for updating CI to test both numpy 1 and 2? If not, can that be added? At the moment, I don't see an indication that we pass tests with 2.0, and this seems like an important distinction to be defensive about in CI

[qiyunzhu]

@wasade NumPy 2.0 hasn't been released yet, but one can install it by pip install numpy --pre or pip install numpy==2.0.0rc1. I have played around it and found some ways to set up a local test environment. It is feasible that we add this to CI. However, I think that an easier way is that we wait until NumPy 2.0 and relevant dependencies (like Pandas and h5py) in their compatible versions are released. This will make the CI much simpler.

[qiyunzhu]

Looks like h5py 3.11.0 was just released. This is the main unblocker. Now we need to wait for the release of Pandas 2.2.2.

[wasade]

Okay, thanks! So I think then this PR is "draft" NumPy 2.0 support but we cannot encode as such in CI. Could a changelog note be added along those lines, nothing likely 2.0 support but not yet part of our testing matrix?

[qiyunzhu]

@wasade Sounds good! Changelog updated.

[wasade]

Thanks!