Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve extract performance via ignoring directories early during os.walk #968

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

akx
Copy link
Member

@akx akx commented Feb 7, 2023

Closes #694 (rebases and reimplements it)

@codecov
Copy link

codecov bot commented Feb 7, 2023

Codecov Report

Attention: Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.

Project coverage is 91.31%. Comparing base (8e10fb5) to head (c68ca01).

Files with missing lines Patch % Lines
babel/messages/extract.py 92.85% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #968      +/-   ##
==========================================
- Coverage   91.31%   91.31%   -0.01%     
==========================================
  Files          27       27              
  Lines        4653     4661       +8     
==========================================
+ Hits         4249     4256       +7     
- Misses        404      405       +1     
Flag Coverage Δ
macos-14-3.10 90.32% <92.85%> (-0.01%) ⬇️
macos-14-3.11 90.25% <92.85%> (-0.01%) ⬇️
macos-14-3.12 90.47% <92.85%> (-0.01%) ⬇️
macos-14-3.13 90.47% <92.85%> (-0.01%) ⬇️
macos-14-3.8 90.18% <92.85%> (-0.01%) ⬇️
macos-14-3.9 90.25% <92.85%> (-0.01%) ⬇️
macos-14-pypy3.10 90.32% <92.85%> (-0.01%) ⬇️
ubuntu-24.04-3.10 90.34% <92.85%> (-0.01%) ⬇️
ubuntu-24.04-3.11 90.28% <92.85%> (-0.01%) ⬇️
ubuntu-24.04-3.12 90.49% <92.85%> (-0.01%) ⬇️
ubuntu-24.04-3.13 90.49% <92.85%> (-0.01%) ⬇️
ubuntu-24.04-3.8 90.21% <92.85%> (-0.01%) ⬇️
ubuntu-24.04-3.9 90.27% <92.85%> (-0.01%) ⬇️
ubuntu-24.04-pypy3.10 90.34% <92.85%> (-0.01%) ⬇️
windows-2022-3.10 90.35% <92.85%> (-0.01%) ⬇️
windows-2022-3.11 90.29% <92.85%> (-0.01%) ⬇️
windows-2022-3.12 90.50% <92.85%> (-0.01%) ⬇️
windows-2022-3.13 90.50% <92.85%> (-0.01%) ⬇️
windows-2022-3.8 90.32% <92.85%> (-0.01%) ⬇️
windows-2022-3.9 90.28% <92.85%> (-0.01%) ⬇️
windows-2022-pypy3.10 90.35% <92.85%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@akx akx marked this pull request as ready for review February 16, 2023 19:10
@akx
Copy link
Member Author

akx commented Feb 20, 2023

@DenverCoder1 If you have a moment, could you review this? :)

@DenverCoder1
Copy link
Contributor

Looks good to me (I haven't tested)

I noticed a similar method in frontend.py -

def _make_directory_filter(ignore_patterns):
"""
Build a directory_filter function based on a list of ignore patterns.
"""
def cli_directory_filter(dirname):
basename = os.path.basename(dirname)
return not any(
fnmatch.fnmatch(basename, ignore_pattern)
for ignore_pattern
in ignore_patterns
)
return cli_directory_filter
- it seems to not need to create relative paths there, but maybe it's different. Possibly some logic could be combined or renamed to be consistent, but besides that, it seems like this should work.

@akx akx marked this pull request as draft February 28, 2023 06:59
@akx akx marked this pull request as ready for review January 14, 2025 15:18
@akx akx requested a review from tomasr8 January 14, 2025 15:19
@@ -111,10 +111,25 @@ def _strip(line: str):
comments[:] = map(_strip, comments)


def default_directory_filter(dirpath: str | os.PathLike[str]) -> bool:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this function be considered part of the public API? If so, shouldn't we deprecate it first?


for pattern, method in method_map:
if method == "ignore" and pathmatch(pattern, dir_rel):
return False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line's missing test coverage ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants