Using IterableDataset with Torch DataLoader throws error.

[LeenaShekhar]

Hi,

I have a general question regarding using Dataset in Streaming mode – Is IterableDataset not to be used with Pytorch DataLoader? I can use Dataset with the DataLoader without any issues (as is also mentioned in the examples), but I cannot do so with the former. I am quite new to the HF Dataset library so my apologies if this is already mentioned somewhere (I am still looking).

I get the following error, which makes sense because this is streaming mode, but I am unclear about how to design so that I can do batching then:

File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py”, line 67, in iter
return iter(range(len(self.data_source)))
TypeError: object of type ‘IterableDataset’ has no len()


Any help is appreciated. Thank you.

I have asked the same question here: https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/5

[cccntu]

PyTorch only recognize torch.utils.data.IterableDataset.
You can use this:

torch_iter_dataset = hf_iter_dataset.with_format("torch")

Or you can customize with something like this:

import torch

class TorchIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, hf_iter_ds):
        super().__init__()
        self.ds = hf_iter_ds

    def __iter__(self):
        for ex in self.ds:
            yield ex

Reference:

datasets/src/datasets/iterable_dataset.py

Lines 480 to 494 in c722810

	import torch
	class TorchIterableDataset(IterableDataset, torch.utils.data.IterableDataset):
	pass
	cls = TorchIterableDataset
	else:
	cls = IterableDataset
	return cls(
	ex_iterable=ex_iterable,
	info=info,
	split=split,
	format_type=format_type,
	shuffling=shuffling,
	)

[LeenaShekhar]

Thank you for the response. Yes, I was given a similar answer here: #2583.