PYO3_PYTHON and LD_LIBRARY_PATH pointing to venv are ignored

[jakob-lilliemarck]

Bug Description

I would like pyo3 to use the .venv I've created in my python library, when calling functions from that library from rust. - but it ignores the values configured in my environment.

From what I've read I understood it as PYO3_PYTHON and/or LD_LIBRARY_PATH can be used to configure the path to the python interpreter executable. I exported them in the shell and then ran my rust project like so

export PYO3_PYTHON=/path/to/my/python/project/.venv/bin/python
export LD_LIBRARY_PATH=/path/to/my/python/project/.venv/bin/python
cargo run --bin fxctl ping

ping is just a function I use to test the setup. In the rust app, the ping function is invoked through a clap command. The handler looks like so:

use pyo3::prelude::*;

pub fn ping() -> PyResult<()> {
    Python::with_gil(|py| {
        let sys = PyModule::import(py, "sys").unwrap();

        let py_path: String = sys.getattr("executable").unwrap().extract().unwrap();
        let py_version: String = sys.getattr("version").unwrap().extract().unwrap();

        println!("path: {}\nversion: {}", py_path, py_version);
    });
    Ok(())
}

Steps to Reproduce

Reproduction repository:
https://github.com/jakob-lilliemarck/pyo3-python-env-ignored

Backtrace

There is no backtrace as there is no actual error

Your operating system and version

Linux 6.6.65-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Dec 11 22:24:04 UTC 2024 x86_64 GNU/Linux

Your Python version (python --version)

3.12.7 (main, Oct 1 2024, 11:15:50) [GCC 14.2.1 20240910]

Your Rust version (rustc --version)

rustc 1.85.0-nightly (a224f3807 2024-12-09)

Your PyO3 version

0.23.3

How did you install python? Did you use a virtualenv?

Can't recall, but from the looks of it I installed the global one using brew, then I created a virtual env in my project by python -m venv .venv

Additional Info

No response

[Hennzau]

Hi,
I’m not sure if it’s a typo, but the export of PYO3_PYTHON should use one =:

export PYO3_PYTHON=/path/to/my/python/project/.venv/bin/python

Regarding LD_LIBRARY_PATH, it should point to the directory containing libpython.so. This is typically not found in the .venv but rather in the global installation, for example:

/home/linuxbrew/.linuxbrew/opt/[email protected]/lib

My workaround is to run cargo build/run after activating my venv and i add the ld path of my global python (see #4813)

[jakob-lilliemarck]

@Hennzau thanks for your reply. That's indeed a typo, I edited it to avoid confusion. However the issue persists, I've set up a simple reproduction repository. See the README for steps to install and run it.
https://github.com/jakob-lilliemarck/pyo3-python-env-ignored

Exporting PYO3_PYTHON in the terminal before running with cargo seems to make no difference. However, if I activate the venv prior to running the rust program then the path is correct. . .venv/bin/python.

I also tried to call a shell in the main.rs to see if I could activate the virtualenv from Rust, while I got exitcode 0, that did not change the executable path that PyO3 discovered.

My expectation would be that I could point PyO3 to a virtualenv and that it should then activate and use that virtualenv.

[Hennzau]

Hi, the easiest solution is to always activate your venv before running cargo build/run.

Once pyo3 is compiled you cannot change anything from your main.rs, the only thing you can try is doing all your things in a build.rs script.

If you don't want to activate your venv:

I tried something with the PYO3_CONFIG_FILE env variable, it compiles successfully but I can't run it (it fails). The idea is to create a config file:

implementation=CPython
version=3.13
shared=true
abi3=false
lib_name=python3.13
lib_dir=/home/USER/path/to/lib/cpython-3.13.1-linux-x86_64-gnu/lib
executable=/home/USER/path/to/pyo3-python-env-ignored/.venv/bin/python
pointer_width=64
build_flags=
suppress_build_script_link_lines=false

and pass it while running cargo build:

PYO3_CONFIG_FILE=$(pwd)/pyo3_config_file cargo build

It should build successfully but then when i run it, I get:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ec07a612180 (most recent call first):
  <no Python frame>

[jakob-lilliemarck]

Once pyo3 is compiled you cannot change anything from your main.rs, the only thing you can try is doing all your things in a build.rs script.

hmm 🤔 I was under the impression that the python binary was linked at runtime, not during compilation..? Clearly I don't really understand how it works. The docs state the following:

PyO3 uses a build script (backed by the pyo3-build-config crate) to determine the Python version and set the correct linker arguments. By default it will attempt to use the following in order:

Any active Python virtualenv.
The python executable (if it's a Python 3 interpreter).
The python3 executable.

You can override the Python interpreter by setting the PYO3_PYTHON environment variable, e.g. PYO3_PYTHON=python3.7, PYO3_PYTHON=/usr/bin/python3.9, or even a PyPy interpreter PYO3_PYTHON=pypy3.

reference: https://pyo3.rs/main/building-and-distribution

Reading that it seems that configuring PYO3_PYTHON should indeed take precedence over the default order. However it is not really clear to me if PYO3_PYTHON variable is intended to be used during buildtime or runtime.

The use case I have is that I want to write a rust server application (with tokio) and then have that server application invoke functions in a python library. Having to activate the venv before starting the rust server doesn't seem ideal.

While calling a python library from Rust may seem overly complicated, I was hoping to be able to do some machine learning in my python lib (using np and keras) while keeping everything database- and infrastructure-related on the Rust side.

[jakob-lilliemarck]

Alright, turns out this isn't a bug at all 🫣 please feel free to close this ticket.

PYO3 seems to do exactly what can be expected of it, the issue was due to my lack of understanding of the python interpreter and virtual environments.

However, the pyo3 documentation referenced above was not really helpful in clarifying what was required to configure pyo3 to use a specific virutal env at runtime. That could be a potential enhancement. The documentaiton mentions LD_LIBRARY_PATH for unix systems, and PATH for windows, but PATH really seems to be to only variable that is actually required at runtime, even on unix (I'm using Manjaro Linux). Also, the use-case for the mentioned PYO3_PYTHON variable is still unclear to me. My inital understand was that it intended to be used at runtime to point to a python environment, but now I think it seems to be used at build time, though I am not really sure. I don't think the documentation is really clear on that either 🤔

By modifying the PATH variable it is indeed possible to configure which virtualenv to use, at runtime, from the Rust program that embeds python. I've updated the reproduction repository with a working (but crude) example. Modifying environment variables from Rust requires a single unsafe call on unix though, so that's something to be aware of.