Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle client timeout when running CompleteMultipartUpload #110

Open
JustinKyleJames opened this issue Mar 7, 2024 · 1 comment
Open
Milestone

Comments

@JustinKyleJames
Copy link
Contributor

When CompleteMultipartUpload is executed on a very large file, the client may timeout and retry resulting in some confusing errors.

Here is the flow:

  1. Client calls CompleteMultipartUpload.
  2. The part files are streamed from the S3 API to iRODS in parallel.
  3. The client is not sending or receiving data so it can have a socket timeout.
  4. When it times out, the client may retry the CompleteMultipartUpload.
  5. On the second call to CompleteMultipartUpload, the S3 API will attempt to open the data object. This will fail because the data object is in intermediate state as the previous uploads are still executing.
  6. The client reports an internal server error.
  7. The S3 API reports an error like the following: "Failed open data stream to iRODS - path=/tempZone/home/alice/alice-bucket/1Gfile".
  8. iRODS reports an error about a missing replica token.
@JustinKyleJames
Copy link
Contributor Author

Among the considerations, we might should prevent multiple simultaneous calls to CompleteMultipartUpload with the same upload id. That way we can send a more useful error message to the client.

@korydraughn korydraughn added this to the 0.3.0 milestone Mar 7, 2024
@trel trel modified the milestones: 0.3.0, 0.4.0 Aug 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants