You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, all S3 bucket syncs treat the entire bucket like a flat directory. While this is the nature of S3 buckets, treating "/" characters as individual "sub-folders" in the bucket could massively improve performance. The Minio.list_objects call in the S3 bucket task specifies recursive=True:
This should probably be False, but that would require a lot of other changes.
Additionally, this would greatly improve the potential implementation of #282. As it stands, a query to hold all of the data objects under the target collection is required. This would mean that the entire S3 bucket is being held in memory (possibly - depends on the implementation of Minio.list_objects) and the entire target collection's contents as well, which could potentially be very large.
The text was updated successfully, but these errors were encountered:
Currently, all S3 bucket syncs treat the entire bucket like a flat directory. While this is the nature of S3 buckets, treating "/" characters as individual "sub-folders" in the bucket could massively improve performance. The
Minio.list_objects
call in the S3 bucket task specifiesrecursive=True
:irods_capability_automated_ingest/irods_capability_automated_ingest/tasks/s3_bucket_sync.py
Line 122 in ec34cb1
This should probably be
False
, but that would require a lot of other changes.Additionally, this would greatly improve the potential implementation of #282. As it stands, a query to hold all of the data objects under the target collection is required. This would mean that the entire S3 bucket is being held in memory (possibly - depends on the implementation of
Minio.list_objects
) and the entire target collection's contents as well, which could potentially be very large.The text was updated successfully, but these errors were encountered: