Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove new Asset Manager asset files from NFS once they have been uploaded to S3 #323

Closed
floehopper opened this issue Dec 5, 2017 · 6 comments
Assignees

Comments

@floehopper
Copy link
Contributor

floehopper commented Dec 5, 2017

All Asset Manager assets are now stored and served from S3, so there is no longer any need to store the asset files on NFS as well as S3. The asset files will still need to be saved to NFS on upload so that the Sidekiq workers (which might be on a different machine) can access the file for virus scanning and S3 upload. However, once the file has been uploaded to S3, we should be able to delete the file from NFS.

Note that we'll need to be careful that methods on the Asset class which rely on an underlying file being present continue to work in a sensible fashion when the underlying file has been removed.

Note also that we'll also need to remove asset files for existing assets after we've done this.

@floehopper floehopper self-assigned this Dec 5, 2017
@floehopper floehopper changed the title Remove new asset's file from NFS once it has been uploaded to S3 Remove new Asset Manager asset files from NFS once they have been uploaded to S3 Dec 5, 2017
@floehopper
Copy link
Contributor Author

floehopper commented Dec 6, 2017

#328 is somewhat relevant to this issue.

@floehopper
Copy link
Contributor Author

#328 has now been merged.

@floehopper
Copy link
Contributor Author

As a precursor to this I'm currently looking at extending the examples in spec/requests/virus_scanning_spec.rb to include the upload to S3. This will give me more confidence that nothing is broken when we start deleting files from NFS after the S3 upload.

As a precursor to the above, I'm looking at making it possible to use the fake S3 functionality within the specs. And in order to do this I'm first going to address a problem with the configuration of this which I highlighted in this comment.

@floehopper
Copy link
Contributor Author

I'm also planning to address this bug as part of the above work: #281

@floehopper
Copy link
Contributor Author

I've opened #373 to do this.

@floehopper
Copy link
Contributor Author

#373 has now been deployed to integration, staging & production and so we are now deleting files from NFS for new assets once their files have been uploaded to S3. This PR also included a Rake task which will make it possible to implement #296. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant