Skip reading files with incorrect extension #318
Conversation
Signed-off-by: Sarah Yurick <[email protected]>
|
We might need to expand the list of extensions since some files are format like |
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
|
|
||
| input_extensions = {os.path.splitext(f)[-1] for f in input_files} | ||
| if len(input_extensions) != 1: | ||
| raise RuntimeError( |
There was a problem hiding this comment.
An example of when we would expect this RuntimeError is for:
doc = DocumentDataset.read_json(in_files)
Where in_files is a string path to a directory with multiple JSONL files and a CRC file. Since the CRC file is not explicitly being filtered out, we raise the error.
There was a problem hiding this comment.
We can leave this as is for now.
In theory there might be cases where a user filters by [.json, .jsonl] using the file filter, but will raise errors here. In practice I expect it to be unlikely so we can wait an see if there is any user feedback around this.
nemo_curator/utils/file_utils.py
Outdated
| root: str, | ||
| recurse_subdirectories: bool = True, | ||
| followlinks: bool = False, | ||
| filter_by: Optional[Union[str, List[str]]] = None, |
There was a problem hiding this comment.
All of these examples work:
(1)
input_files = get_all_files_paths_under(in_files, filter_by="jsonl")
input_dataset = DocumentDataset.read_json(input_files)
(2)
input_files = get_all_files_paths_under(in_files, filter_by=["jsonl"])
input_dataset = DocumentDataset.read_json(input_files)
(3)
# Returns a list containing only .jsonl, .parquet, and .csv files
input_files = get_all_files_paths_under(in_files, filter_by=["jsonl", "parquet", "csv"])
nemo_curator/utils/file_utils.py
Outdated
| if file.endswith(tuple(file_extensions)): | ||
| filtered_files.append(file) | ||
| else: | ||
| warnings.warn(f"Skipping read for file: {file}") |
There was a problem hiding this comment.
I wonder if this might get too noisy in some cases. I'm leaning towards warning once if we have to skip, but not for every file we skip.
|
|
||
| input_extensions = {os.path.splitext(f)[-1] for f in input_files} | ||
| if len(input_extensions) != 1: | ||
| raise RuntimeError( |
There was a problem hiding this comment.
We can leave this as is for now.
In theory there might be cases where a user filters by [.json, .jsonl] using the file filter, but will raise errors here. In practice I expect it to be unlikely so we can wait an see if there is any user feedback around this.
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
|
Thanks @ayushdg ! Updated. |
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
|
Thank you @praateekmahajan ! I have addressed all your comments. |
Signed-off-by: Sarah Yurick <[email protected]>
Closes #214.