[SPIKE] Research methods of identifying unused dependencies

[exalate-issue-sync]

Research automated options for identifying unused dependencies in requirements.txt

https://github.com/matthewhughes934/py-unused-deps seems like the type of thing we’re looking for but it’s low usage and contributor numbers take it out of the running in my book.

Upon completion of research, create follow-up tickets for actionable findings.

QA Notes

null

DEV Notes

null

Design

null

See full ticket and images here: FECFILE-1844

[exalate-issue-sync]

Sasha Dresden commented: I think I found a solution that works for this: [https://deptry.com/|https://deptry.com/|smart-link]
This project seems to be under active and continual maintenance unlike many of the others I looked into which hadn’t been touched in over a year. My one concern with it is that the latest release is 0.21.1 so it hasn’t done a 1.0 release yet. However it worked really well for what I did with it.

The nice thing about it in comparison to the others I looked at was that it was highly configurable from the {{pyproject.toml}} file. So for example when I ran I was getting a lot of false positives complaining about {{fecfiler}} but I was able to add that to the {{known_first_party}} and it would essentially ignore that issue. I could also tell it to ignore other rules as needed. For example, since the only place we use gunicorn is in the {{dockerfile }}and {{run-api.sh}}, which are done without a direct import call so it didn’t catch the usage of it. That type of situation was the hardest to capture with all of these tools, but deptry provided a simple way to ignore these as known requirements.
I was also able to get this to play nicely with circleci. I created a new task in the test job to run this tasks, right after it runs the linter. I ran it with the current requirements.txt file which led to a failed test run, and also ran with the updated requirements.txt file from [https://fecgov.atlassian.net/browse/FECFILE-1776|https://fecgov.atlassian.net/browse/FECFILE-1776|smart-link] which led to a successful test run.

There are some weaknesses to this. It only checks requirements.txt. Well, I had it run requirements-test.txt as a dev requirements, but it wasn’t very good at determining which of those were being used. Going back to the same issues with things like gunicorn or flake8. We don’t use it anywhere in the application, just used as part of the circle test, so it can’t seem to confirm what’s used or not there. It also doesn’t appear to check version, so if you accidentally used django 2 instead of 5 it would mark it as ok because you do have django.

Some others I looked at in addition to the one mentioned in the ticket description:
[https://pypi.org/project/pip-check-reqs/|https://pypi.org/project/pip-check-reqs/|smart-link] last updated Sept 2023

[https://pypi.org/project/pipreqs/|https://pypi.org/project/pipreqs/|smart-link] last updated Feb 2024.

[exalate-issue-sync]

Sasha Dresden commented: Links to passing and failed circle tests for this branch:

Failed:
[https://app.circleci.com/pipelines/github/fecgov/fecfile-web-api/4654/workflows/6fb27307-a3ba-497f-b5be-50cabb1645d5/jobs/13308/parallel-runs/0/steps/0-118|https://app.circleci.com/pipelines/github/fecgov/fecfile-web-api/4654/workflows/6fb27307-a3ba-497f-b5be-50cabb1645d5/jobs/13308/parallel-runs/0/steps/0-118]

Passing:

[https://app.circleci.com/pipelines/github/fecgov/fecfile-web-api/4653/workflows/c45469f7-3774-43de-9cf0-27607198ba1b/jobs/13307/parallel-runs/0/steps/0-118|https://app.circleci.com/pipelines/github/fecgov/fecfile-web-api/4653/workflows/c45469f7-3774-43de-9cf0-27607198ba1b/jobs/13307/parallel-runs/0/steps/0-118]

[exalate-issue-sync]

Todd Lees commented: Solution found and ticket created: [https://fecgov.atlassian.net/browse/FECFILE-1878|https://fecgov.atlassian.net/browse/FECFILE-1878|smart-link]

[exalate-issue-sync]

Shelly Wise commented: No code change or QA review needed on this ticket per DEV.

Moved to Stage Ready.