Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gcloud] fix double decompression when using gzip #1457

Merged
merged 1 commit into from
Jan 11, 2025

Conversation

thuibr
Copy link
Contributor

@thuibr thuibr commented Oct 2, 2024

The self.blob.download_to_file already performs decompression per this StackOverflow post. Rather than attempting to decompress the gzip file a second time and receiving an error, allow self.blob.download_to_file to decompress it.

An alternative to this would be to use the raw_download=True kwarg for self.blob.download_to_file if that is desirable.

This closes #1456.

@thuibr thuibr changed the title prevent automatic decompression by google client Fix Double Decompression When Using GCloud GZIP Oct 2, 2024
@thuibr
Copy link
Contributor Author

thuibr commented Oct 2, 2024

@jschneier what are your thoughts on this change as opposed to using raw_download=True in the client per https://stackoverflow.com/questions/67744979/how-to-prevent-gcs-from-automatically-decompressing-objects-when-using-python-sd ?

@thuibr
Copy link
Contributor Author

thuibr commented Oct 3, 2024

@jschneier I added a test. How do I run the tests in CI?

@thuibr
Copy link
Contributor Author

thuibr commented Oct 7, 2024

@jschneier can you merge this?

@thuibr
Copy link
Contributor Author

thuibr commented Oct 22, 2024

@jschneier can you look at this?

@thuibr thuibr changed the title Fix Double Decompression When Using GCloud GZIP [gcloud] fix double decompression when using gzip Jan 3, 2025
@thuibr
Copy link
Contributor Author

thuibr commented Jan 3, 2025

@jschneier what do I need to do to get this PR looked at and approved?

@jschneier
Copy link
Owner

jschneier commented Jan 4, 2025

@thuibr can you confirm this fixes your issue? We will also need to fix CI.

@thuibr
Copy link
Contributor Author

thuibr commented Jan 6, 2025

@jschneier, I double checked just now that this fixes the issue. I also made a change to the test that I think fixes CI.

@thuibr
Copy link
Contributor Author

thuibr commented Jan 11, 2025

@jschneier do you need to run the CI?

@jschneier jschneier merged commit b79ea31 into jschneier:master Jan 11, 2025
22 checks passed
@thuibr thuibr deleted the gzip-gcs branch January 12, 2025 12:20
@markesha
Copy link
Contributor

@jschneier Do you mind making a release for this change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix GCloud GZIP
3 participants