Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expose dataloader configuration in all configs #1971

Open
felipemello1 opened this issue Nov 8, 2024 · 2 comments
Open

expose dataloader configuration in all configs #1971

felipemello1 opened this issue Nov 8, 2024 · 2 comments
Labels
community help wanted We would love the community's help completing this issue enhancement New feature or request

Comments

@felipemello1
Copy link
Contributor

felipemello1 commented Nov 8, 2024

Currently, we dont allow users to decide on their own the number of works, pin_to_memory, etc. We should expose it in the configs.

Recipe example:

dataloader = DataLoader(

then in the config we have something like:

dataloader:
	batch_size:  ...
	num_workers: 0 # set to -1 to use all. This may not work in some setups.
	pin_memory: False # Set to true to xyz
	#TODO: check if it makes sense to expose other arguments

The script in this PR should help to bulk update the configs: #1954

@felipemello1 felipemello1 added enhancement New feature or request community help wanted We would love the community's help completing this issue labels Nov 8, 2024
@RdoubleA
Copy link
Contributor

RdoubleA commented Nov 8, 2024

cc @andrewkho, we'll likely have a dataloader builder for the torchdata components, and will have to expose this in the config

@andrewkho
Copy link
Contributor

makes sense, definitely in the plan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community help wanted We would love the community's help completing this issue enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants