Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk file operations #62

Open
itsWindows11 opened this issue Jun 22, 2024 · 2 comments
Open

Bulk file operations #62

itsWindows11 opened this issue Jun 22, 2024 · 2 comments

Comments

@itsWindows11
Copy link
Contributor

itsWindows11 commented Jun 22, 2024

Bulk file operations should be standardized in the spec as an optional interface, in which the implementor can efficiently write methods to do item operations like copying and deleting in bulk.

For example if we take the FluentFTP implementation, it has the ability to upload and download files in bulk efficiently. Instead of rendering our abstraction useless in that case, we can use the implementor's underlying capabilities instead of having the consumer deal with each file manually which could be slow and create a lot of network roundtrips.

There are many different implementations for such thing that don't interoperate with each other.

Parallelism when doing bulk operations can also be considered here, but parallelizing single file operations can be done by the consumer manually by opening and writing to the stream, which is mostly outside of this issue's scope.

@itsWindows11
Copy link
Contributor Author

A starter interface I'm proposing would be something like:

namespace OwlCore.Storage;

public interface IBulkOperations : IModifiableFolder
{
    Task<IList<IStorable>> BulkCopyAsync(IEnumerable<IStorable> storablesToCopy, ...);

    Task<IList<IStorable>> BulkMoveAsync(IEnumerable<IStorable> storablesToMove, ...);

    // The storables to delete must be in the folder.
    Task BulkDeleteAsync(IEnumerable<IStorable> storablesToDelete, ...);
}

This can be applied to any IModifiableFolder where doing bulk operations is possible and won't cause any potential API problems when interacting with a web API like Google Drive or OneDrive.

@Arlodotexe
Copy link
Owner

Arlodotexe commented Jun 23, 2024

We should tackle recursion a bit more before we attempt this. Creating a fallback extension method (covering when the interface is not implemented) is again the main concern here.

Similar to the problem in #35 (comment), what would a fallback for bulk operations use for parallelism, if any? We'd instead have to pick whatever is best (parallel or sequential) for the underlying API, which is information only held by implementors or consumers of library.

The issues with recursion would also apply here if these are recursive-- There are too many possible ways to crawl a graph for us to simply expose a 'standard' set of options as a parameter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants