Non-thread-safe use of Py_SET_REFCNT in destructors

[colesbury]

Bug report

When calling finalizers from dealloc, we temporarily increase the refcount with Py_SET_REFCNT and then decrease it again (also with Py_SET_REFCNT). This isn't thread-safe in the free threading build because some other thread may concurrently try to increase the refcount during a racy dictionary or list access.

cpython/Objects/object.c

Lines 552 to 566 in dabcecf

	/* Temporarily resurrect the object. */
	Py_SET_REFCNT(self, 1);
	PyObject_CallFinalizer(self);
	_PyObject_ASSERT_WITH_MSG(self,
	Py_REFCNT(self) > 0,
	"refcount is too small");
	/* Undo the temporary resurrection; can't use DECREF here, it would
	* cause a recursive call. */
	Py_SET_REFCNT(self, Py_REFCNT(self) - 1);
	if (Py_REFCNT(self) == 0) {
	return 0; /* this is the normal path out */
	}

We have similar issues with some watcher events:

cpython/Objects/funcobject.c

Lines 1095 to 1102 in dabcecf

	assert(Py_REFCNT(op) == 0);
	Py_SET_REFCNT(op, 1);
	handle_func_event(PyFunction_EVENT_DESTROY, op, NULL);
	if (Py_REFCNT(op) > 1) {
	Py_SET_REFCNT(op, Py_REFCNT(op) - 1);
	return;
	}
	Py_SET_REFCNT(op, 0);

This can lead to crashes when we have racy accesses to objects that may be concurrently finalized. For example:

./python -m test test_free_threading -m test_racing_load_super_attr -v -F

Test (un)specialization of LOAD_SUPER_ATTR opcode. ... ./Include/internal/pycore_object.h:593: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Enable tracemalloc to get the memory block allocation traceback

<object at 0x200041800f0 is freed>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

(Observed more frequently on macOS, but also occurs on Linux)

In macOS buildbot: https://buildbot.python.org/#/builders/1368/builds/2251/steps/6/logs/stdio

Linked PRs

• gh-127582: Make object resurrection thread-safe for free threading. #127612
• [3.13] gh-127582: Make object resurrection thread-safe for free threading. (GH-127612) #127659