creating dependency groups similarly to poetry

[demetrius-mp]

is there a way to create multiple dependency groups? as far as i know, currently there is only tool.rye.dev-dependencies for development dependencies.

in poetry, for instance, you can create many different groups.

i think that the greatest advantage would be to, for example, separate docs dependencies from testing dependencies, so that CI wont download docs dependencies as it is not required.

[mitsuhiko]

I want to avoid too much customizability which then needs to be replicated by every tool around. So if rye adds some dependency groups it mean that every tool on top needs that too. I quite appreciate that Rust has a relatively simple dev + prod + build setup for these. Can you share a bit more how you would use this though?

[demetrius-mp]

sure, i can provide an example of how this could be used

take for example this project. if you head to the pyproject.toml file, in this line you can see i defined a group named doc, which holds only documentation packages, such as mkdocs.

now suppose i want to set up continuous integration for this package, which will run a formatting/linting tool + tests. we all agree that the CI environment does not need to download the documentation packages, since the docs are built with readthedocs. Hence, one can setup the package installation command like this, which will install every group of dependencies, except for the doc group. This would reduce the execution time of the CI.

[mitsuhiko]

I see. This makes sense. I will give this some further thinking. In the Rust world this all just falls under a dev setup, but there are additional dependencies that can be set in this case. They all are published as well, but that's something you don't want here from what I understand.

I'm not fully opposed to having exception groups but I'm worried about this level of customization as it means that all tools now need different forms of configuration to opt into all these groups. For instance I still would like to allow the editor to trigger a build with for instance rye run build-docs (preferably even just rye docs as a well known command) without having to learn something about how dependencies are set up.

[demetrius-mp]

i see

another option i can think of for creating separate dependency groups is to create them as if they were optional dependencies.

for example, say i want to separate my docs and test, dependencies, i would do something like

[project]
name = "mypackage"
# ...
[project.optional-dependencies]
dev-docs = ["mkdocs==..."]
dev-test = ["pytest==..."]

of course, rye would need to provide some command to install only some of these groups. i believe pdm does exactly what i described.

however, doing this is somewhat a "wrong usage" of the optional dependencies feature.

[mitsuhiko]

I agree that this is not a great way to go about it. In some sense for me the issue here already starts with that tools like mkdocs, tox, black or whatever really don't belong into the same virtualenv as the project itself (usually!). The reason is that the project has it's own dependencies and they can leak into other and conflict. So in some sense all these tools really need a proper isolation from each other.

[demetrius-mp]

completely agree with you!

[majidaldo]

i feel these 'super'/upper configurations should be handled by a proper programming language like nickel. the programming language then generates static configuration to be consumed by rye/poetry/pdm.

[doolio]

I agree that this is not a great way to go about it. In some sense for me the issue here already starts with that tools like mkdocs, tox, black or whatever really don't belong into the same virtualenv as the project itself (usually!). The reason is that the project has it's own dependencies and they can leak into other and conflict. So in some sense all these tools really need a proper isolation from each other.

And this is exactly what hatch does.