Add support for GraalPy

[msimacek]

Description

Hi, I'm a GraalPy developer and I'd like to add support for GraalPy to pybind11. We've been using pybind11 with GraalPy for quite a while, but we relied on a downstream patch. Most of the changes to non-test code just add another condition to existing ifdefs that were there for PyPy. In order to pass the tests on GraalPy, I had to skip several tests that rely on reference counting and finalizers, because unfortunately it's not really possible to always reliably trigger the GC-dependent finalizers at a specific point on GraalPy. I also marked a few failing tests with xfail for now if they seemed like bugs that should be fixed on GraalPy's side. I'll try to look into fixing those in the near future. If you're not opposed, I'd also like to try adding current version of GraalPy to the CI matrix.

CC @timfel @steve-s

Suggested changelog entry:

Added support for GraalPy Python implementation (https://github.com/oracle/graalpython)

[henryiii]

CI addition sounds good, thanks!

[msimacek]

Hi @rwgk, I finally got the CI working. The issue with the "unstable" (maximum internals version) tests was because in GraalPy PyCapsule_GetName currently returns a copy of the name and pybind11 does pointer comparison with the original string it passed in when creating the capsule. I'll fix that for the next release of GraalPy, but for now, I disabled using the capsule names on GraalPy (like this). Let me know if that's ok. I could also do a full string comparison, but that sounds like it could kill performance. Other than that, I tried to "localize" the skipping of GC/refcount-dependent test parts, as you requested.

[rwgk]

What do you think about using:

if env.GRAALPY:
pytest.skip("ConstructorStats is incompatible with GraalPy.")

here (and in similar places elsehwere)? — Then we don't have to indent the entire block below.

[henryiii]

That will report that it skipped the whole test, when half of it did run. If you do that, then you could just put it as a decorator and actually skip the whole test. Maybe it's possible to split this in half?

[rwgk]

I just tried out my suggestion locally. The pytest output looks fine to me (ignoring the existing print noise):

test_buffers.py ................ss.........### test_submodule_buffers(module_&)::Matrix @ 0x3175a00 destroyed 2x3 matrix
### test_submodule_buffers(module_&)::Matrix @ 0x3175b80 destroyed 5x4 matrix
                                                                                                [100%]

============================================================ short test summary info =============================================================
SKIPPED [1] test_buffers.py:86: ConstructorStats is incompatible with GraalPy.
SKIPPED [1] test_buffers.py:125: ConstructorStats is incompatible with GraalPy.
========================================================= 25 passed, 2 skipped in 0.02s ==========================================================

That will report that it skipped the whole test

What makes you think that?

The main motivation for my suggestion was that the diff will be less noisy:

diff --git a/tests/test_buffers.py b/tests/test_buffers.py
index 535f33c2..8f0c849e 100644
--- a/tests/test_buffers.py
+++ b/tests/test_buffers.py
@@ -82,6 +82,9 @@ def test_from_python():
         for j in range(m4.cols()):
             assert m3[i, j] == m4[i, j]
 
+    if env.GRAALPY:
+        pytest.skip("ConstructorStats is incompatible with GraalPy.")
+
     cstats = ConstructorStats.get(m.Matrix)
     assert cstats.alive() == 1
     del m3, m4
@@ -118,6 +121,9 @@ def test_to_python():
     mat2[2, 3] = 5
     assert mat2[2, 3] == 5
 
+    if env.GRAALPY:
+        pytest.skip("ConstructorStats is incompatible with GraalPy.")
+
     cstats = ConstructorStats.get(m.Matrix)
     assert cstats.alive() == 1
     del mat

Maybe it's possible to split this in half?

I recommend against doing that in this PR.

[msimacek]

Done (except a couple of methods where the refcount checks were in the middle of the method)

[rwgk]

Nice, thanks!