Parsing of dependencies from different build systems

[jugmac00]

Describe the bug
run pyscan

❯ pyscan 
pyscan v0.1.5 | by Aswin S (github.com/aswinnnn)
Using pyproject.toml as source...
thread 'main' panicked at 'no entry found for key', src/parser/extractor.rs:61:24
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

To Reproduce
run pyscan on any project

Expected behavior
should work

Desktop (please complete the following information):

• OS: Ubuntu 20.04
• rust: 1.70

[aswinnnn]

can you show me the pyproject.toml that pyscan detects? like the content inside?

[jugmac00]

Oh, does this imply that only projects with a full pyproject.toml are supported?

I am using setuptools with setup.py and setup.cfg and the pyproject.toml only for configuring the build backend a la...

[build-system]
requires = ["setuptools>=46.4.0", "wheel"]
build-backend = "setuptools.build_meta"

e.g. https://github.com/jugmac00/flask-reuploaded

It would be great to have a helpful error message 👍

[aswinnnn]

I see. Pyscan currently looks for the [dependencies] table on a pyproject.toml which seems to be the common way to convey dependencies, I'm not sure how exactly setuptools differs in specifying deps but this certainly seems interesting enough to be added support for.

[aswinnnn]

https://peps.python.org/pep-0631/

Here's the PEP from which the parser for pyproject.toml is partially based on.

[jugmac00]

the common way to convey dependencies

I would not call it the "common way" - maybe it is the suggested way, but it is certainly not followed by all packaging tools.

You need to know that the lowest common denominator is that projects configure the build backend in the pyproject.toml, and from here on, it entirely depends on the build system how meta data is configured.

• The currently probably still most widely used tool is setuptools, which has even its own configuration files - ie the config lives outside the pyproject.toml.
• Poetry uses pyproject.toml, but stores the dependencies in tool.poetry.dependencies (and other keys), see https://python-poetry.org/docs/managing-dependencies/
• Tools following your idea are probably hatch and flit, maybe others.

So, way to go :-)

P.S.: It is even not mandatory that Python projects use a pyproject.toml.

[aswinnnn]

I see. I was under the assumption that the PEP would be a little bit more popular than i thought. Its very weird that the build tools don't follow the PEP and each seems to have its own way of doing it, though. Looks like expanding the parsing of pyproject.toml is something that needs to be done, glad you pointed it out!

[jugmac00]

Python exists for 30+ years, the pep 621 (which superseded the one you mentioned) was only accepted at the end of 2020 (and even only as provisional, see https://discuss.python.org/t/pep-621-round-3/5472/109 ) - so it will take time until most package managers will follow that, and probably a good part of Python projects won't update to use a modern package manager for a very long time.

[aswinnnn]

So support for the setuptools way of dependency spec in pyproject.toml is underway. Do you have any suggestions for other build systems which pyscan should support parsing from? My knowledge regarding them is limited

[jugmac00]

The most common ones I encounter are:

• setuptools
• poetry
• hatch
• flit
• pdm

[aswinnnn]

great, pyscan should be able to support them by the release of next version, thanks

[aswinnnn]

Looking at https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#declaring-required-dependency

Looks like setuptools does follow the way pyscan scans for dependencies, but since you're using setup.py like in flask-reuploaded all the dependency spec goes into install_requires, so pyscan would need to implement a way to parse that as well. Looks like setup.py needs its own parse implementation.