3.13 freethreaded deadlock

[RCoder01]

Why does this program deadlock with python version 3.13t and feature auto-initialize?

Python::with_gil(|py| {
    thread::spawn(|| {
        Python::with_gil(|py| {});
    }).join();
});

My understanding of freethreaded python is that multiple threads should be able to attach to the python interpreter at once, so I would expect this program to succeed. But in this example if a thread spawned while being attached tries to attach, it causes the program to deadlock. Is this expected behavior? Or an issue somewhere?

[davidhewitt]

Can you give more details about how you have set up your environment to use 3.13t? Can you validate a free-threaded interpreter is being used? The GIL may also be re-enabled if an incompatible module is loaded; have you tried with PYTHON_GIL=0 (if it is definitely a free-threaded interpreter)?

[RCoder01]

I'm on windows with 3.13t installed from clicking the checkbox in the python 3.13 installer.

Running this script:

use std::{sync::atomic::{AtomicUsize, Ordering}, thread};

use pyo3::{types::PyAnyMethods, Python};

fn main() {
    let python = std::env::var("PYO3_PYTHON").unwrap();
    let gil = std::env::var("PYTHON_GIL").unwrap();
    println!("PYO3_PYTHON={python}");
    println!("PYTHON_GIL={gil}");
    Python::with_gil(|py| {
        print_interpreter(py);

        spawn_threads();
        // py.allow_threads(spawn_threads);
    })
}

fn print_interpreter(py: Python<'_>) {
    let sys = py.import("sys").unwrap();
    let version: String = sys.getattr("version").unwrap().extract().unwrap();
    println!("{}", version);
}

fn spawn_threads() {
    let counter = AtomicUsize::new(0);
    thread::scope(|t| {
        t.spawn(|| thread_fn(&counter));
        t.spawn(|| thread_fn(&counter));
    })
}

fn thread_fn(counter: &AtomicUsize) {
    Python::with_gil(|_py| {
        counter.fetch_add(1, Ordering::SeqCst);
        // This should only work if both threads are running concurrently
        while counter.load(Ordering::SeqCst) != 2 {
            continue;
        }
        println!("Thread {:?} finished", thread::current().id());
    });
}

cargo.toml:

[package]
name = "pyo3free"
version = "0.1.0"
edition = "2021"

[dependencies]
pyo3 = { version = "0.23.2", features = ["auto-initialize"] }

Hangs after printing this:

C:\pyo3free>cargo run
   Compiling pyo3free v0.1.0 (C:\pyo3free)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.43s
     Running `target\debug\pyo3free.exe`
PYO3_PYTHON=C:\Users\[name]\AppData\Local\Programs\Python\Python313\python3.13t.exe
PYTHON_GIL=0
3.13.0 experimental free-threading build (tags/v3.13.0:60403a5, Oct  7 2024, 09:53:29) [MSC v.1941 64 bit (AMD64)]

If I uncomment the py.allow_threads and remove the line before, it prints this and exits normally:

C:\pyo3free>cargo run
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.14s
     Running `target\debug\pyo3free.exe`
PYO3_PYTHON=C:\Users\[name]\AppData\Local\Programs\Python\Python313\python3.13t.exe
PYTHON_GIL=0
3.13.0 experimental free-threading build (tags/v3.13.0:60403a5, Oct  7 2024, 09:53:29) [MSC v.1941 64 bit (AMD64)]
Thread ThreadId(2) finished
Thread ThreadId(3) finished

[davidhewitt]

Thanks, I can reproduce, and minimised to this smaller program:

use std::thread;

use pyo3::Python;

fn main() {
    Python::with_gil(|py| {
        // comment out the line below to reproduce the deadlock
        py.allow_threads(|| std::thread::spawn(|| Python::with_gil(|_| {})).join());
        spawn_threads();
    })
}

fn spawn_threads() {
    let barrier = std::sync::Barrier::new(2);
    // using a barrier requires both threads to attach to the Python
    // runtime concurrently, this program can only complete on the free-threaded build
    thread::scope(|t| {
        t.spawn(|| Python::with_gil(|_| barrier.wait()));
        t.spawn(|| Python::with_gil(|_| barrier.wait()));
    })
}

Looking in a debugger, it looks like the deadlock is in new_threadstate -> _PyGC_ImmortalizeDeferredObjects -> _PyEval_StopTheWorld.

It seems that by adding the py.allow_threads() line to spawn a new thread-state first, this deadlock doesn't happen.

cc @ngoldbaum @colesbury - I guess there's something special about the first non-main thread spawned? If so, to avoid this deadlock maybe PyO3 could internally always spawn a short lived dummy thread like this as part of the initial initialization to avoid users hitting this case.

[colesbury]

This is expected behavior. In the original example, you need py.allow_threads around the join() call. In @davidhewitt's example, you need a py.allow_threads around the barrier.wait().

There's a brief description about this in the C API "howto". You need to use the GIL state APIs in the same places as you would use them in the default (with GIL) build.

Although free threading allows multiple threads to be attached to the Python VM at once, there are some times where a single thread requires exclusive access (a stop-the-world pause). Here are a few examples:

• During garbage collection in order to get a globally consistent view of reference counts and references between objects
• In 3.13t (but not 3.14t): when the first background thread is started in order to mark certain objects as immortal
• When sys.settrace and sys.setprofile is called in order to instrument running code objects and threads
• Before os.fork() is called in order to get the VM in a more consistent-ish state before forking
• When certain attributes on functions objects are modified

[davidhewitt]

Thanks, that makes sense. If this behaviour is 3.13t specific and goes away on 3.14 then it probably does make sense for PyO3 to have the "spawn a dummy background thread" hack just so that complexity stays away from users. I'll push a patch and a test.

[colesbury]

No, that's the opposite of what I intended to get across. Please don't start dummy background threads. You will not fix the deadlocks that way. You need the py.allow_threads in the appropriate places.

[davidhewitt]

Ok, sure thing 👍

[ngoldbaum]

I think the documentation probably needs better advice about exactly when py.allow_threads is needed and also to mention that deadlocks are possible because the interpreter can trigger global synchronization events for various reasons.

I'll try to tackle that this week.

[alex]

A related question for me is what is the overhead of `py.allow_threads()` in a free-threaded build. We know that in GIL builds it can be quite substantial ( pyca/cryptography#11900 (comment))

…

On Mon, Dec 2, 2024 at 1:25 PM Nathan Goldbaum ***@***.***> wrote: I think the documentation probably needs better advice about exactly when py.allow_threads is needed and also to mention that deadlocks are possible because the interpreter can trigger global synchronization events for various reasons. I'll try to tackle that this week. — Reply to this email directly, view it on GitHub <#4738 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBCQVJXFEZZQJ6BVS3D2DSQZ5AVCNFSM6AAAAABSTUZGUSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNBTHE4TEMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[colesbury]

On my M1 mac it's about 12 nanoseconds per Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS pair. It's not really directly comparable to the cryptography benchmark, which is showing overheads in the 10s of microseconds (i.e., 1000x) for 64 bytes, but that's due to lock contention. If there's no contention, then the GIL-enabled build takes ~38 nanoseconds per pair.

(I suspect we could also make the GIL behave much better when there's lock contention and the GIL is released only for short sections)