venv base path does not resolve symlinks using realpath() #106045
Comments
|
I've run into this as well: In my case, it was on ubuntu 22.04, the venv could not be created with a failure to reproducer in GHA: https://github.com/mayeut/sandbox/actions/runs/7518304095/job/20465487706#step:3:81 |
|
Note that resolving symlinks will break Homebrew, see astral-sh/uv#1640 |
Do you have instructions how to reproduce this? We'd like to avoid this problem in uv. From Then
Same questions, do you have instructions how to reproduce this (outside the github actions one)? I tried this with a python checkout (4d3ee77): ./configure --prefix=/opt/cpython1 --enable-optimizations
make -s -j32
sudo make installThen i tried: mkdir a
cd a
ln -s /opt/cpython1/bin/python3 python3
cd ..
mkdir b
cd b
../a/python3 -m venv venv
venv/bin/python -c "import struct; import subprocess"
cd ..
mkdir c
cd c
virtualenv -p ../a/python3 venv
venv/bin/python -c "import struct; import subprocess"But this all passed. |
|
On ubuntu 22.04, install I think the fact that the symlink is in |
|
Do you have more information about where the python3.11 came from? I still can't reproduce FROM ubuntu:22.04
RUN apt update
RUN apt-get install -y python3-distutils
ENV DEBIAN_FRONTEND=noninteractive
RUN apt install -yy git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
WORKDIR /root
RUN git clone https://github.com/python/cpython/
WORKDIR /root/cpython
RUN git checkout v3.11.8
RUN ./configure --prefix=/opt/cpython1
RUN make -s -j32
RUN make install
WORKDIR /root
RUN /opt/cpython1/bin/python3 --version
RUN ln -s /opt/cpython1/bin/python3 /usr/local/bin/python3
RUN /usr/local/bin/python3 --version
RUN /usr/local/bin/python3 -c "import struct; import subprocess"
RUN /usr/local/bin/python3 -m venv venv1
RUN venv1/bin/python -c "import struct; import subprocess"
RUN venv1/bin/python -m venv venv2
RUN venv2/bin/python -c "import struct; import subprocess" |
|
I could reproduce it using your dockerfile with a slight modification. |
|
To summarize, the error only happens (editing this to 3.13 to be actionable on main) when you have a custom built python that is linked to exactly I've tried to trace the cause and it seems that This in turn changes I unfortunately don't understand enough about the python sysconfig machinery to understand how/why Note that for the system (ubuntu |
|
I haven't looked into this in detail, but might the real issue here be that you're effectively running the interpreter in a broken system-wide environment under
|
I don't know what a "broken system-wide environment" means. As I said in the description, I think this is a bug in venv since it doesn't match what getpath.c does. Rather than having venv have complicated logic to figure out what "home" should be, perhaps it should just run something like: I've worked around the issue by creating wrapper shell scripts for executables, rather than using symlinks, e.g. |
|
In order not to break usage involving symlinks to a python install tree, rather than resolving the realpath, only executable symlinks can be checked. xref pypa/virtualenv#2770 (comment) |
Yes, I've been tracking this down, and it boils down to Line 653 in 6ea04da
I note also that setting home in I've spent a lot of time digging into the logic of
Clearly if the executable (symlinked) is in a different location to the prefix, then the documented approach is going to result in a computed prefix from the symlink (if that prefix has the appropriate landmark "lib-dynlib" directory) This explains why resolving the symlink to executable before using that as Note that today, CPython has no problem with non-venv symlinked executables living outside of the prefix. Using exactly the same setup as above, we get the desired behaviour: Perhaps @FFY00 (given their recent changes to getpath) has some idea of whether we can use exactly the same logic for venv Fundamentally, my belief is that the definition of Given that today it works well to run a symlinked executable which is outside of its prefix for non-venv cases, I would propose that the PEP is updated to replace I'd be happy to turn this into a discussion on python.org, if this is the best way to establish a solid rationale and consensus (please confirm if this is the best approach). |
|
To follow-up to my very long last message - I think following what CPython is doing for prefix resolution (for non-venvs) at startup would work well for all cases that I've seen. Namely, it uses a non-path segment recursive symlink resolve on the executable, which then is then used to search for the stdlib. This would require no change in It turns out (after a lot of searching), that this is a two-line fix to diff --git a/Modules/getpath.py b/Modules/getpath.py
index c34101e7208..e82ba23c4c2 100644
--- a/Modules/getpath.py
+++ b/Modules/getpath.py
@@ -412,6 +412,9 @@ def search_up(prefix, *landmarks, test=isfile):
if isfile(candidate):
base_executable = candidate
break
+ if base_executable:
+ # Update the executable directory to be based on the resolved base executable
+ executable_dir = basename(base_executable)
break
else:
# We didn't find a 'home' key in pyvenv.cfg (no break), reset venv_prefix.I'll dig into how to write a test for this, and submit. One thing to note: I think the calculation of |
Yeah, it's known that the
Please submit it! I think we can treat it as a bug, so we can still backport the fix to 3.12 and 3.13.
With GH-127972 and GH-127974, on POSIX builds with a shared libpython, this should now be mitigated by looking at the libpython directory before resorting to the base interpreter, which should be much more reliable. |
It does not seem to be helping in this case (on macOS): With #115237 resolving |
It seems there is a bug in venv (and a similar one in virtualenv) where the "base" path in pyvenv.cfg is set incorrectly. If Python is installed in a non-standard folder, e.g. /usr/local/python-X.Y.Z and then symlinked into /usr/local/bin/python3, the venv package does not work correctly. I believe the source of the trouble is the setting of "home" variable. Specifically, this line:
The
dirnameresult is used to set "home". If the executable path is /usr/local/bin/python3 (actually a symlink to /usr/local/python-X.Y.Z/bin/python3), the "home" should not be set to /usr/local. Changing the above line (in theensure_directories()) function to:This fixes the problem. I believe this is consistent with what the getpath.py module in Python does.
I noticed with problem when running the most recent Debian OS, version 12. It includes Python 3.11 and therefore /usr/lib/python3.11 exists. With the above bug, the venv tries to use /usr/lib/python3.11 as the sys.path. Importing the
structmodule fails with the mysterious error:Linked PRs
venvcreation from a python executable symlink #115237The text was updated successfully, but these errors were encountered: