Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[python-package] UNKNOWN python package #6665

Closed
ZhenyiLin opened this issue Oct 7, 2024 · 5 comments · May be fixed by #6681
Closed

[python-package] UNKNOWN python package #6665

ZhenyiLin opened this issue Oct 7, 2024 · 5 comments · May be fixed by #6681
Assignees
@jameslamb
Labels
bug

Comments

@ZhenyiLin
Copy link

ZhenyiLin commented Oct 7, 2024 •
edited
Loading

Description

I have used cmake to precompiled a .so file. But when I use build-python to create python support. It ended up create a unknown-0.0.0 package.

Reproducible example

sudo apt-get install --no-install-recommends git cmake build-essential libboost-dev libboost-system-dev libboost-filesystem-dev
sudo mkdir -p /mnt/workspace
sudo chown $(whoami):$(whoami) /mnt/workspace
cd /mnt/workspace

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
cmake -B build -S . -DUSE_GPU=1
cmake --build build -j$(nproc)

apt-get install python3-venv
pip install setuptools numpy scipy scikit-learn -U
sudo sh ./build-python.sh install --precompile

Environment info

LightGBM version or commit hash:

Command(s) you used to install LightGBM

root@5567918b09b1:/mnt/workspace/LightGBM# ./build-python.sh install --precompile
building lightgbm
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.10/dist-packages (1.2.2.post1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (1.2.0)
Requirement already satisfied: packaging>=19.1 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (24.1)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (2.0.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
found pre-compiled lib_lightgbm.so
--- building sdist ---
* Creating isolated environment: virtualenv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
running check
creating lightgbm-4.5.0.99
creating lightgbm-4.5.0.99/lightgbm
creating lightgbm-4.5.0.99/lightgbm.egg-info
creating lightgbm-4.5.0.99/lightgbm/lib
copying files to lightgbm-4.5.0.99...
copying LICENSE -> lightgbm-4.5.0.99
copying MANIFEST.in -> lightgbm-4.5.0.99
copying README.rst -> lightgbm-4.5.0.99
copying pyproject.toml -> lightgbm-4.5.0.99
copying lightgbm/__init__.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/basic.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/callback.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/compat.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/dask.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/engine.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/libpath.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/plotting.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/py.typed -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/sklearn.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm.egg-info/PKG-INFO -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/dependency_links.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/requires.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/top_level.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm/lib/lib_lightgbm.so -> lightgbm-4.5.0.99/lightgbm/lib
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
Writing lightgbm-4.5.0.99/setup.cfg
Creating tar archive
removing 'lightgbm-4.5.0.99' (and everything under it)
Successfully built lightgbm-4.5.0.99.tar.gz
--- installing lightgbm ---
Looking in links: .
Processing ./lightgbm-4.5.0.99.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=1794 sha256=31cf85902f1e3ffcb3b79747c9d3629bdf509a4dfe6f2dda9c7bf91c58ce4506
  Stored in directory: /tmp/pip-ephem-wheel-cache-bfyn99ka/wheels/97/48/5c/b2fa3ad0b5ff4da83888744e25e16ce761d05dc33923b317f1
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
  Attempting uninstall: UNKNOWN
    Found existing installation: UNKNOWN 0.0.0
    Uninstalling UNKNOWN-0.0.0:
      Successfully uninstalled UNKNOWN-0.0.0
Successfully installed UNKNOWN-0.0.0

Additional Comments
@jameslamb
Copy link
Collaborator

What Linux distribution are you using? That'd help us to replicate this.

@jameslamb jameslamb added the bug label Oct 7, 2024
@ZhenyiLin
Copy link
Author

ZhenyiLin commented Oct 7, 2024 via email

@jameslamb jameslamb changed the title UNKNOWN python package [python-package] UNKNOWN python package Oct 14, 2024
@jameslamb jameslamb self-assigned this Oct 14, 2024
@jameslamb
Copy link
Collaborator

I was able to reproduce this in an Ubuntu 22.04 container today (on my M2 mac).

docker run \
    --rm \
    -it ubuntu:22.04 \
    bash

apt-get update -y
apt-get install -y --no-install-recommends \
    git \
    ca-certificates \
    curl \
    build-essential \
    libboost-dev \
    libboost-system-dev \
    libboost-filesystem-dev \
    ocl-icd-opencl-dev \
    python3-dev \
    python3-pip

CMAKE_VERSION="3.30.0"
curl -O -L \
    "https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-$(arch).sh"
mkdir /opt/cmake
sh "cmake-${CMAKE_VERSION}-linux-$(arch).sh" --skip-license --prefix=/opt/cmake
ln -sf /opt/cmake/bin/cmake /usr/local/bin/cmake
cmake --version
# cmake version 3.30.0

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
echo "commit: $(git rev-parse HEAD)"
# commit: bbeecc09af946c5ff9b84d1ada4749a9f26bca31

cmake -B build -S . -DUSE_GPU=1
cmake --build build -j$(nproc)

ln -sf /usr/bin/python3 /usr/bin/python

python -m pip install \
    -U \
    numpy \
    scipy \
    scikit-learn \
    setuptools

sh ./build-python.sh install --precompile
logs (click me) 
building lightgbm
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.10/dist-packages (1.2.2.post1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (1.2.0)
Requirement already satisfied: packaging>=19.1 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (24.1)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (2.0.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
found pre-compiled lib_lightgbm.so
--- building sdist ---
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
running check
creating lightgbm-4.5.0.99
creating lightgbm-4.5.0.99/lightgbm
creating lightgbm-4.5.0.99/lightgbm.egg-info
creating lightgbm-4.5.0.99/lightgbm/lib
copying files to lightgbm-4.5.0.99...
copying LICENSE -> lightgbm-4.5.0.99
copying MANIFEST.in -> lightgbm-4.5.0.99
copying README.rst -> lightgbm-4.5.0.99
copying pyproject.toml -> lightgbm-4.5.0.99
copying lightgbm/__init__.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/basic.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/callback.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/compat.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/dask.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/engine.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/libpath.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/plotting.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/py.typed -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/sklearn.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm.egg-info/PKG-INFO -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/dependency_links.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/requires.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/top_level.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm/lib/lib_lightgbm.so -> lightgbm-4.5.0.99/lightgbm/lib
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
Writing lightgbm-4.5.0.99/setup.cfg
Creating tar archive
removing 'lightgbm-4.5.0.99' (and everything under it)
Successfully built lightgbm-4.5.0.99.tar.gz
--- installing lightgbm ---
Looking in links: .
Processing ./lightgbm-4.5.0.99.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=1794 sha256=e88b5b7ae36ef88e896ee2e2e6b4b7964b0f45ae209a1d87efa05cae773a9bff
  Stored in directory: /tmp/pip-ephem-wheel-cache-6jha994c/wheels/1d/09/6b/d983276114c6137d7226d1d09502c590b61e8604505dffb9cf
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
Successfully installed UNKNOWN-0.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
cleaning up

Later, I'll try removing this line so I can inspect the build directory:

LightGBM/build-python.sh

Line 379 in bbeecc0

rm -rf ./lightgbm-python

I suspect this MIGHT be related to #6624. Will provide more updates soon.

@jameslamb
Copy link
Collaborator

I suspect this MIGHT be related to #6624.

Nope, this was not it.

@ZhenyiLin I found that just updating to a more recent pip resolves this. Can you please try that?

pip install --upgrade pip
sh build-python.sh install --precompile
Processing ./lightgbm-4.5.0.99.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: lightgbm
  Building wheel for lightgbm (pyproject.toml) ... done
  Created wheel for lightgbm: filename=lightgbm-4.5.0.99-py3-none-any.whl size=3026623 sha256=cab9fd30930a192c76671c15546ee075bcd7f57c58367ffcda33d8e5b7fc8b0b
  Stored in directory: /tmp/pip-ephem-wheel-cache-832doefo/wheels/1d/09/6b/d983276114c6137d7226d1d09502c590b61e8604505dffb9cf
Successfully built lightgbm
Installing collected packages: lightgbm
  Attempting uninstall: lightgbm
    Found existing installation: lightgbm 4.5.0.99
    Uninstalling lightgbm-4.5.0.99:
      Successfully uninstalled lightgbm-4.5.0.99
Successfully installed lightgbm-4.5.0.99

I saw that work with pip 24.2.

pip --version
# pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)

I'm not sure what range of versions will work.

Root Cause

It looks like python3-pip / python3-venv on Ubuntu 22.04 ships with pip==22.0.2 and setuptools==59.6.0.

setuptools did not add support for pyproject.toml until v61.0.0 (see the setuptools docs).

Your script is running pip install -U setuptools (where -U is an alias for --upgrade), which installs the latest compatible setuptools at /usr/local/lib/python3.10/dist-packages. But I think the system-installed pip at /usr/lib/python3/dist-packages is ignoring that and loading the setuptools at /usr/lib/python3/dist-packages.

pip install -U pip puts a new pip into /usr/local/lib/python3.10/dist-packages, which then uses the newer setuptools there.

What lightgbm can do

We could have build-python.sh write a setup.cfg file when --precompile is passed, so other users won't face this.

Ubuntu 22.04 is only 2.5 years... it won't reach End of Standard Support until April 2027 (https://ubuntu.com/about/release-cycle). I think this is very worth supporting.

jameslamb added a commit that referenced this issue Oct 15, 2024
@jameslamb
[ci] [python-package] support setuptools<61 in build-python.sh (fixes #…
308382c 
…6665)
@jameslamb jameslamb mentioned this issue Oct 15, 2024
Draft
@ZhenyiLin
Copy link
Author

Thanks, updating pip works!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jameslamb jameslamb

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

WIP: [ci] [python-package] support setuptools<61 in build-python.sh (fixes #6665) microsoft/LightGBM
2 participants
@jameslamb @ZhenyiLin