Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add link_only option to remote file sources #19124

Closed
guerler opened this issue Nov 9, 2024 · 1 comment
Closed

Add link_only option to remote file sources #19124

guerler opened this issue Nov 9, 2024 · 1 comment

Comments

@guerler
Copy link
Contributor

guerler commented Nov 9, 2024

Suggesting to introduce an optional link_only boolean parameter in file_sources_conf.yml (default: false). When enabled, datasets are linked rather than copied during uploads. Although the upload/fetch mechanism already supports link_only uploads, this functionality is not currently configurable for file sources. By enabling this feature, users can avoid duplicating large datasets when the remote file source resides on a filesystem already mounted to their compute nodes. Potentially this can also be done using the file source type attribute.

Ping @jmchilton for suggestions and comments. Thanks in advance.

@guerler
Copy link
Contributor Author

guerler commented Nov 15, 2024

Resolved by #19132

@guerler guerler closed this as completed Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant